Most of us get how crucial AI evals are now. The thing is, almost all the eval platforms we’ve seen are clunky – there’s too much manual setup and adaptation needed, which breaks developers’ workflows.

Last week, we released a radically simpler workflow.

You can now connect your GitHub repo to Openlayer, and every commit on GitHub will also commit to Openlayer, triggering your tests. You now have continuous evaluation without extra effort.

You can customize the workflow using our CLI and REST API. We also offer template repositories around common use cases to get you started quickly.

You can leverage the same setup to monitor your live AI systems after you deploy them. It’s just a matter of setting some variables, and your Openlayer tests will run on top of your live data and send alerts if they start failing.

We’re very excited for you to try out this new workflow, and as always, we’re here to help and all feedback is welcome.

New features

• Developer workflow
  • GitHub integration
  • CLI and REST API
  • Sample repositories for various workflows
  • Ability to clone sample repositories in Openlayer UI
• New test: column A grouped by column B

Improvements

• Move test options to header bar in modals
• Improvements to test results modals
• Improve layout of workspace onboarding
• Ability to delete tests
• Relevant tests created automatically upon project creation in onboarding
• Polished design of in-app callouts
• Polish to activity log
• Reorganization of docs
• Allow None values in token column

Bug fixes

• Row outputs in panel are injected into chat history format when they should not be
• Row panel dropdowns do not appear when opened from a test modal
• Monitoring graphs showed no recent results even when there were some
• Opening create test modal for Group by Column test crashed the app
• Column parameters was not able to be changed for Group By tests
• Creating a commit without a model breaks
• Project filtering did not work in overview page
• Creating Character Length tests runs into client-side error when there are no input variables
• Client-side exception when opening requests