It is a visitor weblog submit cowritten with athenahealth.
athenahealth a number one supplier of network-enabled software program and providers for medical teams and well being techniques nationwide. Its digital well being information, income cycle administration, and affected person engagement instruments enable anytime, anyplace entry, driving higher monetary outcomes for its clients and enabling its supplier clients to ship higher high quality care.
Within the synthetic intelligence (AI) house, athenahealth makes use of information science and machine studying (ML) to speed up enterprise processes and supply suggestions, predictions, and insights throughout a number of providers. From its first implementation in automated doc providers, touchlessly processing thousands and thousands of provider-patient paperwork, to its more moderen work in digital assistants and enhancing income cycle efficiency, athenahealth continues to use AI to assist drive effectivity, service capabilities, and higher outcomes for suppliers and their sufferers.
This weblog submit demonstrates how athenahealth makes use of Kubeflow on AWS (an AWS-specific distribution of Kubeflow) to construct and streamline an end-to-end information science workflow that preserves important tooling, optimizes operational effectivity, will increase information scientist productiveness, and units the stage for extending their ML capabilities extra simply.
Kubeflow is the open-source ML platform devoted to creating deployments of ML workflows on Kubernetes easy, moveable, and scalable. Kubeflow achieves this by incorporating related open-source instruments that combine properly with Kubernetes. A few of these initiatives embody Argo for pipeline orchestration, Istio for service mesh, Jupyter for notebooks, Spark, TensorBoard, and Katib. Kubeflow Pipelines helps construct and deploy moveable, scalable ML workflows that may embody steps like information extraction, preprocessing, mannequin coaching, and mannequin analysis within the type of repeatable pipelines.
AWS is contributing to the open-source Kubeflow neighborhood by offering its personal Kubeflow distribution (known as Kubeflow on AWS) that helps organizations like athenahealth construct extremely dependable, safe, moveable, and scalable ML workflows with diminished operational overhead by integration with AWS managed providers. AWS offers numerous Kubeflow deployment choices like deployment with Amazon Cognito, deployment with Amazon Relational Database Service (Amazon RDS) and Amazon Easy Storage Service (Amazon S3), and vanilla deployment. For particulars on service integration and out there add-ons for every of those choices, seek advice from Deployment.
In the present day, Kubeflow on AWS offers a transparent path to utilizing Kubeflow, augmented with the next AWS providers:
Many AWS clients are benefiting from the Kubeflow on AWS distribution, together with athenahealth.
Right here, the athenahealth MLOps workforce talk about the challenges they encountered and the options they created of their Kubeflow journey.
Challenges with the earlier ML surroundings
Previous to our adoption of Kubeflow on AWS, our information scientists used a standardized set of instruments and a course of that allowed flexibility within the expertise and workflow used to coach a given mannequin. Instance elements of the standardized tooling embody a knowledge ingestion API, safety scanning instruments, the CI/CD pipeline constructed and maintained by one other workforce inside athenahealth, and a typical serving platform constructed and maintained by the MLOps workforce. Nonetheless, as our use of AI and ML matured, the number of instruments and infrastructure created for every mannequin grew. Though we had been nonetheless capable of help the prevailing course of, we noticed the next challenges on the horizon:
- Upkeep and progress – Reproducing and sustaining mannequin coaching environments took extra effort because the variety of deployed fashions elevated. Every undertaking maintained detailed documentation that outlined how every script was used to construct the ultimate mannequin. In lots of instances, this was an elaborate course of involving 5 to 10 scripts with a number of outputs every. These needed to be manually tracked with detailed directions on how every output could be utilized in subsequent processes. Sustaining this over time turned cumbersome. Furthermore, because the initiatives turned extra advanced, the variety of instruments additionally elevated. For instance, most fashions utilized Spark and TensorFlow with GPUs, which required a bigger number of surroundings configurations. Over time, customers would swap to newer variations of instruments of their growth environments however then couldn’t run older scripts when these variations turned incompatible. Consequently, sustaining and augmenting older initiatives required extra engineering effort and time. As well as, as new information scientists joined the workforce, data transfers and onboarding took extra time, as a result of synchronizing native environments included many undocumented dependencies. Switching between initiatives confronted the identical points as a result of every mannequin had its personal workflows.
- Safety – We take safety critically, and subsequently prioritize compliance with all contractual, authorized, and regulatory obligations related to ML and information science. Information have to be utilized, saved, and accessed in particular methods, and now we have embedded sturdy processes to make sure our practices adjust to our authorized obligations in addition to align with trade greatest practices. Previous to Kubeflow adoption, guaranteeing that information was saved and accessed in a particular approach concerned common verification throughout a number of, various workflows. We knew that we might enhance efficiencies by consolidating these various workflows onto a single platform. Nonetheless, that platform would must be versatile sufficient to combine properly with our standardized tooling.
- Operations – We additionally noticed a chance to extend operational effectivity and administration by centralizing the logging and monitoring of the workflows. As a result of every workforce had developed their very own instruments, we collected this info from every workflow individually and aggregated them.
The info science workforce evaluated numerous options for consolidating the workflows. Along with addressing these necessities, we seemed for an answer that may combine seamlessly with the prevailing standardized infrastructure and instruments. We chosen Amazon EKS and Kubeflow on AWS as our workflow answer.
The info scientist growth cycle incorporating Kubeflow
An information science undertaking begins with a clear slate: no information, no code, solely the enterprise drawback that may be solved with ML. The primary activity is a proof of idea (POC) to find if the info holds sufficient sign to make an ML mannequin efficient at fixing the enterprise drawback, beginning with querying for the uncooked dataset from our Snowflake information warehouse. This stage is iterative, and the info scientists use Kubernetes pods or Kubeflow Jupyter notebooks throughout this course of.
Our Kubeflow cluster makes use of the Karpenter cluster autoscaler, which makes spinning up assets straightforward for information scientists as a result of they solely have to give attention to defining the specified occasion sorts, whereas the provisioning work is finished by a set of predefined Karpenter provisioners. Now we have separate provisioners for CPU and GPU occasion sorts, and all of the situations supported by Amazon EKS fall in one among these two classes as per our provisioner configuration. The info scientists select occasion sorts utilizing node selectors, and Karpenter takes care of node lifecycle administration.
After the question is developed, the info scientists extract the uncooked information to a location on Amazon S3, then launch a Jupyter pocket book from the AWS Kubeflow UI to discover the info. The aim is to create the function set that can be used to coach the primary mannequin. This enables the info scientists to find out if there may be sufficient sign within the information to meet the client’s enterprise want.
After the outcomes are passable, the info scientists transfer to the subsequent stage of the event cycle and switch their discoveries into a sturdy pipeline. They convert the POC code into production-quality code that runs at scale. To make sure compliance by utilizing permitted libraries, a container is created with the suitable base Docker picture. For our information scientists, now we have discovered that offering a normal Python, TensorFlow, and Spark base picture offers ample flexibility for many, if not all, workloads. They’ll then use the Dockerfile of their part to additional customise their growth surroundings. This Dockerfile is then utilized by the CI/CD course of to construct the elements picture that can be utilized in manufacturing, subsequently sustaining consistency between growth and manufacturing environments.
Now we have a instrument that provides information scientists the flexibility to launch their growth surroundings in a pod operating on Kubernetes. When this pod is operating, the info scientists can then connect the Visible Studio Code IDE on to the pod and debug their mannequin code. After they’ve the code operating efficiently, they’ll then push their adjustments to git and a brand new growth surroundings is created with the latest adjustments.
The usual information science pipeline consists of levels that embody extraction, preprocessing, coaching, and analysis. Every stage within the pipeline seems as a part in Kubeflow, which consists of a Kubernetes pod that runs a command with some info handed in as parameters. These parameters can both be static values or references to output from a earlier part. The Docker picture used within the pod is constructed from the CI/CD course of. Particulars on this course of seem within the CI/CD workflow mentioned within the subsequent part.
Growth Cycle on Kubeflow. The event workflow begins on the left with the POC. The finished mannequin is deployed to the athenahealth mannequin serving platform operating on Amazon ECS.
CI/CD course of supporting automated workflows
As a part of our CI/CD course of, we use Jenkins to construct and take a look at all Kubeflow part photos in parallel. On profitable completion, the pipeline part template comprises reference tips to the photographs, and the ensuing pipeline is uploaded to Kubeflow. Parameters within the Jenkins pipeline enable customers to launch the pipelines and run their mannequin coaching checks after profitable builds.
Alternatively, to keep up a brief growth cycle, information scientists can even launch the pipeline from their native machine, modifying any pipeline parameters they might be experimenting with.
Tooling exists to make sure the reference pointers from the CI/CD construct are utilized by default. If there’s a deployable artifact within the repo, then the CI/CD logic will proceed to deploy the artifact to the athenahealth mannequin serving platform (the Prediction Service) operating on Amazon ECS with AWS Fargate. In spite of everything these levels have handed, the info scientist merges the code to the first department. The pipelines and deployable artifacts are then pushed to manufacturing.
CI/CD Deployment workflow. This diagram describes the Information Science construct and deployment workflow. The CI/CD course of is pushed by Jenkins.
Safety
In consolidating our information science workflows, we had been capable of centralize our strategy to securing the coaching pipeline. On this part, we talk about our strategy to information and cluster safety.
Information safety
Information safety is of the utmost significance at athenahealth. Because of this, we develop and preserve infrastructure that’s absolutely compliant with the rules and requirements that defend the safety and integrity of those information.
To make sure we meet information compliance requirements, we provision our AWS infrastructure in accordance with our athenahealth enterprise pointers. The 2 predominant shops for information are Amazon RDS for extremely scalable pipeline metadata and Amazon S3 for pipeline and mannequin artifacts. For Amazon S3, we make sure the buckets are encrypted, HTTPS endpoints are enforced, and the bucket insurance policies and AWS Id and Entry Administration (IAM) roles comply with the ideas of least privilege when allowing entry to the info. That is true for Amazon RDS information as properly: encryption is at all times enabled, and the safety teams and credential entry comply with the precept of least privilege. This standardization ensures that solely approved events have entry to the info, and this entry is tracked.
Along with these measures, the platform additionally undergoes safety menace assessments and steady safety and compliance scans.
We additionally tackle information retention necessities through information lifecycle administration for all S3 buckets that include delicate information. This coverage robotically strikes information to Amazon S3 Glacier after 30 days of creation. Exceptions to this are managed by information retrieval requests and are permitted or denied on a case-by-case foundation. This ensures that every one workflows adjust to the info retention coverage. This additionally solves the issue with recovering information if a mannequin performs poorly, and retraining is required, or when a brand new mannequin have to be evaluated towards a historic iteration of an older mannequin’s dataset.
For limiting entry to Amazon S3 and Amazon RDS from inside Kubeflow on AWS and Amazon EKS, we use IRSA (IAM Roles for Service Accounts), which offers IAM-based permission provisioning for assets inside Kubernetes. Every tenant in Kubeflow has a novel pre-created service account which we bind to an IAM function created particularly to meet the tenant entry necessities. Person entry to tenants can be restricted utilizing the Amazon Cognito person swimming pools group membership for every person. When a person is authenticated to the cluster, the generated token comprises group claims, and Kubernetes RBAC makes use of this info to permit or deny entry to a selected useful resource within the cluster. This setup is defined in additional element within the subsequent part.
Cluster safety utilizing multi-user isolation
As we famous within the earlier part, information scientists carry out exploratory information analyses, run information analytics, and practice ML fashions. To allocate assets, manage information, and handle workflows primarily based on initiatives, Kubeflow on AWS offers isolation primarily based on Kubernetes namespaces. This isolation works for interacting with the Kubeflow UI; nevertheless, it doesn’t present any tooling to regulate entry to the Kubernetes API utilizing Kubectl. Which means that person entry will be managed on the Kubeflow UI however not over the Kubernetes API through Kubectl.
The structure described within the following diagram addresses this challenge by unifying entry to initiatives in Kubeflow primarily based on group memberships. To realize this, we took benefit of the Kubeflow on AWS manifests, which have integration with Amazon Cognito person swimming pools. As well as, we use Kubernetes role-based entry management (RBAC) to regulate authorization inside the cluster. The person permissions are provisioned primarily based on Amazon Cognito group membership. This info is handed to the cluster with the token generated by the OIDC shopper. This course of is simplified because of the built-in Amazon EKS performance that enables associating OIDC identification suppliers to authenticate with the cluster.
By default, Amazon EKS authentication is carried out by the IAM authenticator, which is a instrument that allows authenticating with an EKS cluster utilizing IAM credentials. This authentication technique has its deserves; nevertheless, it’s not appropriate for our use case as a result of athenahealth makes use of Microsoft Azure Energetic Listing for identification service throughout the group.
Kubernetes namespace isolation. Information Scientists can get hold of membership to a single or a number of teams as wanted for his or her work. Entry is reviewed frequently and eliminated as acceptable.
Azure Energetic Listing, being an enterprise-wide identification service, is the supply of reality for controlling person entry to the Kubeflow cluster. The setup for this consists of creating an Azure Enterprise Utility that acts as service principal and including teams for numerous tenants that require entry to the cluster. This setup on Azure is mirrored in Amazon Cognito by organising a federated OIDC identification supplier that outsources authentication accountability to Azure. The entry to Azure teams is managed by SailPoint IdentityIQ, which sends entry requests to the undertaking proprietor to permit or deny as acceptable. Within the Amazon Cognito person pool, two utility shoppers are created: one is used to arrange the authentication for the Kubernetes cluster utilizing the OIDC identification supplier, and the opposite to safe Kubeflow authentication into the Kubeflow UI. These shoppers are configured to cross group claims upon authentication with the cluster, and these group claims are used alongside RBAC to arrange authorization inside the cluster.
Kubernetes RBAC function bindings are arrange between teams and the cluster function Kubeflow-edit, which is created upon putting in Kubeflow within the cluster. This function binding ensures any person interacting with the cluster after logging in through OIDC can entry the namespaces they’ve permissions for as outlined of their group claims. Though this works for customers interacting with the cluster utilizing Kubectl, the Kubeflow UI at the moment doesn’t provision entry to customers primarily based on group membership as a result of it doesn’t use RBAC. As an alternative, it makes use of the Istio Authorization Coverage useful resource to regulate entry for customers. To beat this problem, we developed a customized controller that synchronizes customers by polling Amazon Cognito teams and provides or removes corresponding function bindings for every person relatively than by group. This setup allows customers to have the identical degree of permissions when interacting with each the Kubeflow UI and Kubectl.
Operational effectivity
On this part, we talk about how we took benefit of the open supply and AWS instruments out there to us to handle and debug our workflows in addition to to attenuate the operational influence of upgrading Kubeflow.
Logging and monitoring
For logging, we make the most of FluentD to push all our container logs to Amazon OpenSearch Service and system metrics to Prometheus. We then use Kibana and the Grafana UI for looking out and filtering logs and metrics. The next diagram describes how we set this up.
Kubeflow Logging. We use each Grafana UI and Kibana to view and sift by the logs
The next screenshot is a Kibana UI view from our pipeline.
Pattern Kibana UI View. Kibana permits for personalized views.
Secure Kubeflow cluster upgrades
As we onboard customers to Kubeflow on AWS, we preserve a dependable and constant person expertise whereas permitting the MLOps workforce to remain agile with releasing and integrating new options. On the floor, Kustomize appears modular for us to allow working and upgrading one part at a time with out impacting others, thereby permitting us so as to add new capabilities with minimal disruption to the customers. Nonetheless, in apply there are situations the place the perfect strategy is to easily spin up a brand new Kubernetes cluster relatively than making use of component-level upgrades for current clusters. We discovered two use instances the place it made extra sense to create fully new clusters:
- Upgrading to a Kubernetes model the place AWS does present in-place cluster upgrades. Nonetheless, it turns into troublesome to check if every of the Kubeflow and Kubernetes assets are working as supposed and the manifests retain backward compatibility.
- Upgrading Kubeflow to a more moderen launch the place there are a number of options added or modified and it nearly at all times is just not a promising concept to carry out in-place upgrades on an current Kubernetes cluster.
In addressing this challenge, we developed a method that allows us to have protected cluster replacements with out impacting any current workloads. To realize this, we needed to meet the next standards:
- Separate the Kubeflow storage and compute assets in order that the pipeline metadata, pipeline artifacts, and person information are retained when deprovisioning the older cluster
- Combine with Kubeflow on AWS manifests in order that when a Kubeflow model improve happens, minimal adjustments are required
- Have an easy technique to roll again if issues go flawed after cluster improve
- Have a easy interface to advertise a candidate cluster to manufacturing
The next diagram illustrates this structure.
Secure Kubeflow Cluster Improve. As soon as testing of the Kubeflow Candidate is profitable, it’s promoted to Kubeflow Prod by an replace to Route 53.
Kubeflow on AWS manifests come pre-packaged with Amazon RDS and Amazon S3 integrations. With these managed providers appearing as widespread information shops, we will arrange a blue-green deployment technique. To realize this, we ensured that the pipeline metadata is continued in Amazon RDS, which works independently of the EKS cluster, and the pipeline logs and artifacts are continued in Amazon S3. Along with pipeline metadata and artifacts, we additionally arrange FluentD to route pod logs to Amazon OpenSearch Service.
This ensures that the storage layer is totally separated from the compute layer and thereby allows testing adjustments throughout Kubeflow model updates on a very new EKS cluster. After all of the checks are profitable, we’re capable of merely change the Amazon Route 53 DNS file to the candidate cluster internet hosting Kubeflow. Additionally, we preserve the outdated cluster operating as backup for just a few days simply in case we have to roll again.
Advantages of Amazon EKS and Kubeflow on AWS for our ML pipeline
Amazon EKS and the Kubeflow on AWS bundle moved our growth workflow to a sample that strongly encourages repeatable mannequin coaching. These instruments enable us to have absolutely outlined clusters with absolutely outlined tenants and run absolutely outlined code.
Lots of wins from constructing this platform are much less quantitative and have extra to do with how the workflows have improved for each platform builders and customers. For instance, MinIO was changed with direct entry to Amazon S3, which strikes us nearer to our authentic workflows and reduces the variety of providers we should preserve. We’re additionally capable of make the most of Amazon RDS because the backend for Kubeflow, which allows simpler migrations between clusters and offers us the flexibility to again up our pipelines nightly.
We additionally discovered the enhancements within the Kubeflow integration with AWS managed providers useful. For instance, with Amazon RDS, Amazon S3, and Amazon Cognito preconfigured within the Kubeflow on AWS manifests, we save effort and time updating to newer distributions of Kubeflow. Once we used to change the official Kubeflow manifests manually, updating to a brand new model would take a number of weeks, from design to testing.
Switching to Amazon EKS offers us the chance to outline our cluster in Kustomize (now a part of Kubectl) and Terraform. It seems that for platform work, Kubernetes and Terraform are very straightforward to work with after placing in sufficient time to be taught. After many iterations, the instruments out there to us make it very straightforward to carry out normal platform operations like upgrading a part or swapping out a complete growth cluster. In comparison with operating jobs off uncooked Amazon Elastic Compute Cloud (Amazon EC2) situations, it’s onerous to match what an infinite distinction it makes to have well-defined pods with assured useful resource cleanup and retry mechanisms inbuilt.
Kubernetes offers nice safety requirements, and now we have solely scratched the floor of what the multi-user isolation permits us to do. We see multi-user isolation as a sample that has extra payoff sooner or later when the coaching platform produces production-level information, and we carry on builders from exterior our workforce.
In the meantime, Kubeflow permits us to have reproducible mannequin coaching. Even with the identical information, no coaching produces an identical fashions, however now we have the subsequent smartest thing. With Kubeflow, we all know precisely what code and information had been used to coach a mannequin. Onboarding has vastly improved as a result of every step in our pipeline is clearly and programmatically outlined. When new information scientists have the duty of fixing a bug, they want a lot much less handholding as a result of there’s a clear construction to how outputs of code are used between levels.
Utilizing Kubeflow additionally yields plenty of efficiency enhancements in comparison with operating on a single EC2 occasion. Usually in mannequin coaching, information scientists want completely different instruments and optimizations for preprocessing and coaching. For instance, preprocessing is commonly run utilizing distributed information processing instruments, like Spark, whereas coaching is commonly run utilizing GPU situations. With Kubeflow pipelines, they’ll specify completely different occasion sorts for various levels within the pipeline. This enables them to make use of the highly effective GPU situations in a single stage and a fleet of smaller machines for distributed processing in one other stage. Additionally, as a result of Kubeflow pipelines describe the dependencies between levels, the pipelines can run levels in parallel.
Lastly, as a result of we created a course of for including tenants to the cluster, there may be now a extra formal technique to register groups to a tenant on the cluster. As a result of we use Kubecost to trace prices in our EKS cluster, it permits us to attribute price to a single undertaking relatively than having price attributed on the account degree, which incorporates all information science initiatives. Kubecost presents a report of the cash spent per namespace, which is tightly coupled to the tenant or workforce that’s chargeable for operating the pipeline.
Regardless of all the advantages, we’d warning to solely undertake this type of migration if there may be whole buy-in from customers. Customers that put within the time get plenty of advantages from utilizing Amazon EKS and Kubernetes, however there’s a vital studying curve.
Conclusion
With the implementation of the Kubeflow on AWS pipeline in our end-to-end ML infrastructure, we had been capable of consolidate and standardize our information science workflows whereas retaining our important tooling (akin to CI/CD and mannequin serving). Our information scientists can now transfer between initiatives primarily based on this workflow with out the overhead of studying how you can preserve a very completely different toolset. For a few of our fashions, we had been additionally pleasantly shocked on the pace of the brand new workflow (5 occasions quicker), which allowed for extra coaching iterations and consequently producing fashions with higher predictions.
We even have established a strong basis to enhance our MLOps capabilities and scale the quantity and dimension of our initiatives. For instance, as we harden our governance posture in mannequin lineage and monitoring, now we have diminished our focus from over 15 workflows to only one. And when the Log4shell vulnerability got here to gentle in late 2021, we had been capable of give attention to a single workflow and shortly remediate as wanted (performing Amazon Elastic Container Registry (Amazon ECR) scans, upgrading Amazon OpenSearch Service, updating our tooling, and extra) with minimal influence to the continued work by the info scientists. As AWS and Kubeflow enhancements turn into out there, we will incorporate them as we see match.
This brings us to an necessary and understated facet of our Kubeflow on AWS adoption. One of many essential outcomes of this journey is the flexibility to roll out upgrades and enhancements to Kubeflow seamlessly for our information scientists. Though we mentioned our strategy to this earlier, we additionally depend on the Kubeflow manifests offered by AWS. We began our Kubeflow journey as a proof of idea in 2019, previous to the discharge of model 1.0.0. (We’re at the moment on 1.4.1, evaluating 1.5. AWS is already engaged on the 1.6 model.) Within the intervening 3 years, there have been not less than six releases with substantial content material. Via their disciplined strategy to integrating and validating these upgrades and releasing the manifests on a predictable, dependable schedule, the Kubeflow workforce at AWS has been essential in enabling the athenahealth MLOps workforce to plan our growth roadmap, and consequently our useful resource allocations and areas of focus, additional into the longer term with better confidence.
You’ll be able to comply with the AWS Labs GitHub repository to trace all AWS contributions to Kubeflow. You can too discover AWS groups on the Kubeflow #AWS Slack Channel; your suggestions there helps AWS prioritize the subsequent options to contribute to the Kubeflow undertaking.
Concerning the authors
Kanwaljit Khurmi is a Senior Options Architect at Amazon Net Providers. He works with the AWS clients to supply steering and technical help serving to them enhance the worth of their options when utilizing AWS. Kanwaljit focuses on serving to clients with containerized and machine studying purposes.
Tyler Kalbach is a Principal Member of Technical Employees at athenahealth. Tyler has roughly 7 years of expertise in Analytics, Information Science, Neural Networks, and growth of Machine Studying purposes within the Healthcare house. He has contributed to a number of Machine Studying options which can be at the moment serving manufacturing visitors. Presently working as a Principal Information Scientist in athenahealth’s Engineering group, Tyler has been a part of the workforce that has constructed the brand new Machine Studying Coaching Platform for athenahealth from the inception of that effort.
Victor Krylov is a Principal Member of Technical Employees at athenahealth. Victor is an engineer and scrum grasp, serving to information scientists construct safe quick machine studying pipelines. In athenahealth he has labored on interfaces, medical ordering, prescriptions, scheduling, analytics and now machine studying. He values cleanly written and properly unit examined code, however has an unhealthy obsession with code one-liners. In his spare time he enjoys listening to podcasts whereas strolling his canine.
Sasank Vemuri is a Lead Member of Technical Employees at athenahealth. He has expertise working with growing information pushed options throughout domains akin to healthcare, insurance coverage and bioinformatics. Sasank at the moment works with designing and growing machine studying coaching and inference platforms on AWS and Kubernetes that assist with coaching and deploying ML options at scale.
Anu Tumkur is an Architect at athenahealth. Anu has over twenty years of structure, design, growth expertise constructing numerous software program merchandise in machine studying, cloud operations, massive information, real-time distributed information pipelines, advert tech, information analytics, social media analytics. Anu at the moment works as an architect in athenahealth’s Product Engineering group on the Machine Studying Platform and Information Pipeline groups.
William Tsen is a Senior Engineering Supervisor at athenahealth. He has over 20 years of engineering management expertise constructing options in healthcare IT, massive information distributed computing, clever optical networks, real-time video modifying techniques, enterprise software program, and group healthcare underwriting. William at the moment leads two superior groups at athenahealth, the Machine Studying Operations and DevOps engineering groups, within the Product Engineering group.
