To start with, you can host your data on a Google Cloud Storage (GCS) or an Amazon Web Services (AWS) S3 bucket, and stream it lazily to your compute instance, on a "need-to-have" basis.
So how does Vaex fit in all of this? There are several substantial benefits to using Vaex as the core technology for building ML solutions even in cloud environments. The Google Cloud AI Platform seen via the Google Cloud Console An important note is that such services are becoming cheaper are more accessible with time. All this means that you can spend more of your time on creating the best model possible, and virtually no time managing, maintaining and configuring your computational resources. In addition, one can submit multiple training jobs which will run independently and not compete with each other for resources. Nowadays, one can "rent" computing instances having few hundreds of vCPUs and several Terabytes of RAM attached, or provision custom cluster configurations.
With this approach computational resources will not be a problem. Therefore, it can be rather convenient, if not necessary, to outsource the heavy lifting stage of ML model creation to a service managed by a recognized cloud provider. On top of that, curating ML models is a continuous process: one will likely need to re-train the model relatively often to account for new data, concept drift, changes in the domain, or simply to improve the model by adjusting the input features, architecture or hyper-parameters. Or you may not even have enough resources at hand to train a sophisticated model on all of the data you have so painstakingly gathered, or use all of the features you have so thoughtfully engineered. Regardless of whether you are using your favorite deep learning framework, trusted gradient boosting machine or a custom ensemble, the model training phase can easily consume most if not all available resources on your laptop or local server, effectively "freezing" your machine and preventing you to do other tasks. Training a machine learning (ML) model is often a rather lengthy and computationally intensive task, especially when using larger quantities of data.
0 kommentar(er)
