We're excited to announce the launch of Modelz beta, a serverless GPU inference platform. Our team has been hard at work building a platform that democratizes access to machine learning, making it easier than ever to build and deploy models for a variety of use cases.

Features

Modelz is a fully managed platform that provides users with a simple API for deploying their machine learning models. The platform takes care of all the underlying infrastructure, including servers, storage, and networking. This means that users can focus on developing their models and deploying them on the platform without worrying about the underlying infrastructure.

Modelz provides the following features out-of-the-box:

• Serverless: Serverless architecture enables us to easily scale up or down according to your needs, allowing us to provide a reliable and scalable solution for deploying and prototyping machine learning applications at any scale.
• Reduce cost: Pay only for the resources you consume, without any additional charges for idle servers or cold starts. Get 30 free minutes of L4 GPU usage when you join us. Attach a payment method and get an extra 90 minutes free.
• OpenAI compatible API: Our platform supports OpenAI compatible API, which means you can easily integrate new open source LLMs into your existing applications with just a few lines of code.
• Prototyping environment: We provide a robust prototyping environment with support for Gradio and Streamlit. With our integration with HuggingFace Space, accessing pre-trained models and launching demos is easier than ever, with just one click. This allows you to quickly test and iterate on your models, saving you time and effort in the development process.

Quick start

Getting started with Modelz is easy and straightforward. Here are the quick steps to get started:

• Sign up for an account on the website.
• Use the Modelz templates to create the deployment.
• Send requests or visit the UI of the deployment.

Here's a full workflow example using the Modelz Beta platform and the bloomz 560M template to create a inference deployment.

After the creation, you will get the detailed information in the UI:

We will show the logs, events (e.g. the deployment scale-up and scale-down events), and the metrics (e.g. total requests, inflight requests) in the dashboard. Besides this, You could get the usage guide too.

Bloomz 560M template is powered by modelz-llm, which provides the OpenAI compatible API for the model. Thus you could use OpenAI python package to use the model. First you need to get the endpoint and API key from the dashboard.

import openai
openai.api_base="https://bloomz-webhiq5i9dagphhu.modelz.io"
# Use your API Key in modelz.
openai.api_key="mzi-xxx"

# create a chat completion
chat_completion = openai.ChatCompletion.create(
  model="any", messages=[{"role": "user", "content": "Hello world"}])

Serverless

The deployment will scale down to 0 after a idle interval (it is configured in the creation page). You could get the autoscaling events and metrics in the dashboard:

Community

Modelz is built on top of envd, mosec, modelz-llm and many other open source projects. If you're interested in joining the Modelz community, here are some ways to get involved:

• Join the Modelz discord community: We have a discord community where you can connect with other developers and data scientists, ask questions, and share your knowledge and expertise.
• Contribute to open source projects: Modelz is built on top of envd, mosec, modelz-llm and many other open source projects. If you're interested in contributing to these projects, you can check out their GitHub repositories and start contributing.
• Share your models and projects: If you've built a machine learning model or a project using Modelz, we'd love to hear about it! You can share your projects on our discord community or on twitter using the hashtag #Modelz, or mention @TensorChord.

As we continue to explore the possibilities of AI and its impact on our world, I wish you a great journey in your pursuit of knowledge and innovation. Whether you are just getting started with AI or are an experienced professional, the field of AI offers endless opportunities for growth and discovery.