Developer Documentation
Welcome to Redwing Vector. This guide provides an overview of our container implementation and code integration process with Sandbox. Our documentation portal is designed to help you understand the functionality of our Vector client library and how to implement vector data types to solve unique business challenges. To provision a license key and customer id please visit https://redwing.ai/plans
- You are running on an arm64 or amd64 compatible machine architecture.
- All M1, M2, and M3 class Apple devices run with arm64
Install Sandbox
This clones and establishes the Sandbox directory from which you will pull and compose the Docker container
git clone https://github.com/redwing-os/sandbox.git
cd sandbox
Set Environment Variables
Create a .env file for Cassandra in current directory which contains your container license key to run the Vector Docker container and enables allotted core utilization within VM
LICENSE_KEY=[your-license-key]
CUSTOMER_ID=[your-customer-id]
DB_HOST=db
DB_PORT=9042
DB_IMAGE=cassandra:latest
GRPC_SERVER=rust_server:50051
RUST_BACKTRACE=full
DB_STARTUP_CMD=""
To enable ScyllaDB your
.env
file must include a DB startup command which gets passed into the
docker-compose.yml during startup
LICENSE_KEY=[your-license-key]
CUSTOMER_ID=[your-customer-id]
DB_HOST=db
DB_PORT=9042
DB_IMAGE=scylladb/scylla:latest
GRPC_SERVER=rust_server:50051
RUST_BACKTRACE=full
DB_STARTUP_CMD="--smp 4 --memory 2G --overprovisioned 1 --listen-address=0.0.0.0 --rpc-address=0.0.0.0 --broadcast-rpc-address=127.0.0.1" # For ScyllaDB Only
Docker Pull
Be sure to have both Docker and docker-compose installed. This step pulls and runs the Docker container from Docker Hub using Sandbox's built-in docker-compose.yml
docker-compose pull && docker-compose up
Cassandra Connect
Establishes Cassandra connection in container to run initialization commands, it will look something similar to sandbox_cassandra_1
docker ps
docker exec -it <your-docker-cassandra-container> cqlsh
Database Configuration
Establishes Cassandra tables and keyspaces for the Vector gRPC service. You can adjust these according to your requirements but keep in mind that the sample scripts and scenarios in Sandbox run according to these keyspace and table name values
CREATE KEYSPACE redwing_keyspace WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};
CREATE TABLE redwing_keyspace.vectors (
key text PRIMARY KEY,
vector list<float>,
created_at timestamp,
updated_at timestamp);
Install Python3 and Pip
Install all required dependencies into your system, this will change based on your runtime environment (Mac/Linux/etc), for Mac be sure to use Homebrew and use brew install python3 make sure you are in the /sandboxdirectory
sudo apt install python3-pip
Install Python Dependencies
Install all required dependencies into your system, make sure you are in the /sandboxdirectory
pip3 install grpcio grpcio-tools sentence-transformers numpy scikit-learn
Run E2E Tests
To run end-to-end tests for all core gRPC functions, make sure you are in the /sandboxdirectory and run the following
python3 test/_test_e2e.py
Explore Catalyst UI
Sandbox comes equipped with our Catalyst UI, which allows you to visualize running containers and infrastructure. To visit Catalyst copy and paste the following url into your browser.
localhost:8080
Run Generated Scenarios
To run all sample data generated scenarios, make sure you are in the /sandboxdirectory and run the following
find ./sample -maxdepth 1 -name "*.py" ! -name "vectordb_pb2*.py" -exec python3 {} \;
Wind Down
To clean up containers and wind down the sandbox environment run the following from the /sandbox directory
docker-compose down
Vector Terraform
Vector Terraform is an auto-provisioning system to fully deploy Redwing Vector to a cloud provider of your choice. To run the app with prompt-based auto-configuration, simply clone this repo and enter the `terraform` directory where the app is installed and run the following command:
git clone https://github.com/redwing-os/terraform.git
cd terraform
sh deploy.sh
To destroy your Terraform-managed infrastructure run the following and enter nothing for all the input prompts except region (i.e. us-east-1)
sudo sh destroy.sh
AWS Credential Configuration
Be sure to run the app from the same terminal window, or ensure the variables are saved to your .bash_profile or .bashrc
Documentation for getting AWS Access Keys is available here, IAM roles must be configured for correct usage and access to Cloudwatch.
AWS IAM DocumentationImportant Notes:
- Never commit your ~/.aws/credentials file to source control.
- Regularly rotate your AWS access keys and always follow best security practices.
- Ensure you have the necessary permissions in AWS to perform the actions required by the application.