Table of Contents
Log ingestor system that can efficiently handle vast volumes of log data, and offer a simple interface for querying this data using full-text search or specific field filters.
- Golang
- Gin
- MongoDB (Atlas)
- HTML (For UI)
- Docker
This is an example of how you may give instructions on setting up your project locally. To get a local copy up and running follow these simple example steps.
Need to have either docker or go installed.
-
Clone the repo
git clone https://github.com/dyte-submissions/november-2023-hiring-pawarpranav83.git
-
Run docker-compose yaml file. It will build the image and run it on port :3000
docker compose up
-
Clone the repo
git clone https://github.com/dyte-submissions/november-2023-hiring-pawarpranav83.git
-
Run
go build .
Perform a POST request with Log data on http://localhost:3000/
Note - The format should be as per the given sample, if any of the fields are nil, the server will give an error. This validation is performed using JSON and binding tags (provided by gin).
It would return an InsertedID, for successful insertion in the database.
HTML UI can be accessed with GET request on http://localhost:3000/
Uses an HTML Form with fields for searching, and on submit, makes a POST request to http://localhost:3000/search, which displays the docs obtained. One field for Full-Text Based Search - Text-Search.
Other fields act as filters.
Created an Aggregation Pipeline -
-
First Stage - Full-Text Based Search on all documents of a collection (created a search index using atlas search).
-
Second Stage - Matching / Filtering the documents obtained from the first stage based on the input filter parameters. Equal to operation, but for message we use the text-based search (created a text index for message), but can only be used when Text-Search field is empty (since MongoDB only allows text-based search on first stage).
-
Third Stage - Limit the number of documents to five.
- Develop a mechanism to ingest logs in the provided format. Performed Validation using struct tags and the ShouldBindWithJSON method provided by Gin.
- Ensure scalability to handle high volumes of logs efficiently. Gin processes requests concurrently, the server can handle high volumes of logs.
- Mitigate potential bottlenecks such as I/O operations, database write speeds, etc. Since only text-based search indexes are added, no bottlenecks in write speeds, and since requests process concurrently so no blocking because of I/O operations.
- Make sure that the logs are ingested via an HTTP server, which runs on port 3000 by default.
- Offer a user interface (Web UI or CLI) for full-text search across logs. Implemented HTML Form for searching logs.
- Filters. Have added filters for all fields and can be used by specifying a value for a field in their corresponding input field.
- Aim for efficient and quick search results. Have search-based indexes that result in a quick text-based search.
- Implement search within specific date ranges. Using gte and lte operators in match stage for timestamp. Also timestamp is stored as value using time.Parse method in golang.
- Allow combining multiple filters. Multiple filters can be combined, if not null those fields are added in the match stage filters. Combined using AND operator.
Pranav Pawar - @pawarpranav83 - [email protected]
Phone Number - 8104273543
- MongoDB Go Driver Docs - https://www.mongodb.com/docs/drivers/go/current/
- Gin Docs - https://github.com/gin-gonic/gin/blob/v1.9.1/docs/doc.md