Run scrapy in docker jobs

Project Details Docker and K8s * Containerize any Microservices based app (Frontend and Backend and DB) and run it as a nonroot user. * Write Kubernetes artefacts to deploy above container on K8s. * Setup Database in Kubernetes cluster. * Deploy microservice to Kubernetes cluster. * Ensure that applications are able to communicate with each other. * Only Ul should be Publicly accessible and not the backend. CI-CD * Automated Application Deployment via CICD. * Setup Jenkins in the Ec2 instance created via Terraform. * Create CICD pipeline to automate the actions done in Phase 1 with stages like Build and Push, push the images to Dockerhub, Update the Kubernetes Artifacts with latest image tag and deploy to the cluster.

I'm looking for a professional who can set up a docker-compose YAML file to effectively deploy and run my FastChat service. This project includes the following requirements: - **Understanding of Docker and docker-compose**: You should have a solid understanding of Docker and docker-compose and how to use them effectively to deploy services. - **Experience with FastChat**: Ideally, you have experience working with FastChat, as the project involves deploying this specific service. - **Setting up Real-time Messaging and User Authentication**: The FastChat service needs to include real-time messaging and user authentication features. - **Knowledge of Docker Compose YAML**: The YAML file should be expertly crafted to ensure the efficient de...

I...task will be setting up and configuring the server for the website. This includes but is not limited to ensuring the server is set up correctly, testing its performance and functionality, and making any necessary adjustments. - Specific Optimization: Experience in optimizing servers for high traffic websites is a must. - Cloudflare Specifics: While the primary focus is on server setup, knowledge of Cloudflare and its CDN (Content Delivery Network) feature is highly desired. Ideal Freelancer: A Cloudflare expert with experience in server setup and optimization, particularly for high traffic websites. Knowledge of setting up and configuring CDNs would be a huge plus. Please only apply if you have previous experience working with Cloudflare and server setup for high ...

We are in the final stage of migrating an existing Django Project to a new machine in a docker compose stack. The database is already moved over. We are stuck with python dependency issues. You need to have good knowledge in - Django and Python3 - Docker, especially Compose stacks - Postgresql You will be working with a system admin who is in the middle of the migration and take over the dependency part, tmate will be used to work together. If you have read this, please add "HDJX34K" to your bid or it will be considered spam, ignored and removed. Your job: - docker compose build - have a look at the python dependency issues - try to resolve them and rebuild. - eventually adjust Django's accordingly. We want this work don...

I having a Application project on vitis that i need to run to see the result on zynq zedboard . Since i dont have the board , im looking for someone to run it and instruct me how to set up the zedboard to run the project. Im working on Windows OS

I am Building AWX Server on AWS EC2 Here is the method I am using: So far, I have: I have already Created an EC2 instance for this server. Cloned the AWX Repo, I have completed all installations required (installed Docker, Docker-Compose, I have used pipx to install the Ansible package) I need help with the rest of the instructions from that link. I am building this server in VS Code Ideal skills for the job include strong understanding of AWS, proficiency in AWX tasks and familiarity with Ubuntu server.

I'm in need of a skilled Python coder to assist me in deploying Locally Hosted Language Model (LLM) applications. Here are the details: - **Operating System**: The deployment is hybrid, so familiarity with Windows, macOS, and Linux is ideal. - **Purpose**: The primary aim of the deployment is for text ebedding (vector db) and generation. - **Requirements**: Specifically, I'm looking for functionality related to Language modeling and retrieval augment generation in the LLM applications. - mongoDB should be set up runining a .sh file - docker-compose deploy of the app Essential skills for this project include a strong background in Python programming and experience with deploying applications on different operating systems. Previous work in...

I need to run TTS wine voices from Linux in docker. I found two tutorials that explain how. Or I need a developer to go through each tutorial and see if you can run the wine gui environment from a docker. Then package that solution as a Docker image I can run locally on lubuntu 22.04. I would do it myself, but I don’t have much experience with docker Requirements: 1 Install balabolka in wine - 2 Install cli balalbolka “bconsole”

I'm seeking a Python developer experienced with web scraping. This job involves setting up a Python script that can scrape a popular review site. Key details to extract: - Review text - Star rating - Date of review - Reviewe...not use captchas, so it should be reasonably straightforward. All reviews for a given provider should be scraped. It would be good to have an option to start with the newest and stop at a certain date. The specific review site can be seen in the sample URL in the attached PDF. I can send further details in a private message including screenshots. Preferred frameworks would be Beautifoulsoup 4 / Scrapy if sufficient and Selenium otherwise. Please state which one you plan to use. Note: it should be possible to run the scrip...

...innovative team. The ideal candidate will have a robust background in software development with more than 7 years of experience, a strong command over various programming languages, and excellent communication skills. Responsibilities: Develop, test, and deploy smart contracts and blockchain-based applications. Design and implement scalable and maintainable microservices architecture. Collaborate with cross-functional teams to deliver high-quality software solutions. Optimize backend processes using Golang for enhanced system performance. Ensure the integration of frontend and backend systems for seamless user experiences. Containerize and orchestrate microservices using Docker and Kubernetes. Skills: Blockchain Proficiency: Expertise in blockchain technologies suc...

I'm seeking a skilled Senior Python Developer to assist with a...on the following tasks: - Extracting data from multiple websites - Automating data extraction and parsing - Analyzing and visualizing scraped data The ideal candidate should have: - Proficiency in Python, with a strong understanding of AI and Machine Learning - Hands-on experience with web scraping, particularly working with diverse data sources - Ability to automate data extraction and processing efficiently - Adept at data analysis, with the ability to visualize and interpret data Although I haven't specified any particular tools or libraries, your familiarity with tools like BeautifulSoup, Scrapy and Selenium would be a bonus. Your ability to recommend and justify the use of specific tools and met...

I'm in need of a proficient individual who can help me set up my NAS System running openSUSE, including the configuration of a Docker system. I already have the hardware for the NAS system, so the primary focus would be on the software setup. Key Requirements: - Installation and configuration of a NAS System on openSUSE - Setting up a Docker system on the NAS - Configuring the NAS to run Web applications, Database services and a Media server - Ensuring the system is capable of supporting connectivity for less than 10 devices Ideal Candidate: - Proficient in setting up NAS Systems on openSUSE - Experienced in Docker system setup and configuration - Knowledgeable in configuring NAS Systems to support various applications - Skille...

I'm looking for someone experienced in running projects from GitHub repositories "" to assist me in setting up and running the demo-k6-operator project on my Mac machine. Your guidance should help me to understand how this project works, which is my primary goal. Key requirements: - Expertise in running and configuring projects from GitHub repositories on Mac machines - Solid understanding of the demo-k6-operator project - Ability to provide clear, detailed instructions and guidance for running the project on my machine - Patience and willingness to explain concepts to help me understand the project better The specific challenge we may face is that I'm using a Mac CPU that is NOT Intel, so experience with this kind

Looking for a Full Stack Developer >6 years of experience in agile development of scalable SaaS apps in the cybersecurity area: Improve our app (URL Risk assessment) Improve our app (Domain Typosquatting) Improve our project (Darknet Monitoring) Create an app "hackerview" for passive attack surfave enumeration Create an app for detecting spoofed social media profiles Create an app to scrape ransomware threat actor website and maitain stats about those sites Improve an existing app to monitor paste sites Mininmal Tech Experience: MERN, REST API (JWT), Django, Python, Github, CI/CD, Celery, Redis, Linux, Postgress, Kubernetes, Docker, Nginx, JS, HTML5, Protocols (DNS, SMTP, SSL, HTTP..).Detailed app specifications will be shared at a later stage. |----...

I am planning to set up a new application on AWS ECS using Docker and I am looking for a skilled expert to assist with the initial setup. Key Project Requirements: - Proficiency setting up and configuring AWS services, particularly ECS and EC2. - Strong understanding and practical experience with Docker. Ideal Skills and Experience: - Extensive experience with AWS ECS and Docker. - Good understanding of best practices for AWS service configuration. - Ability to troubleshoot and provide solutions for potential issues. - Strong communication skills to explain the setup process to me in a clear and understandable way. Please get in touch if you have the skills and experience to support this project.

I'm in search of a full-stack developer to join me in building a scalable, multi-tenant web application. The project requires expertise in Golang and React for back-end and front-end development, respectively. Ideal Candidate: - Proficient in Golang, JavaScript (React) - Skilled in Cassandra, OKTA - Experience with Docker, Kubernetes - Familiarity with Google Cloud Platform -Apply with Github and sample projects The project entails creating a robust, multi-tenant web application that can handle multiple users with varying requirements. It is imperative that the selected developer is competent in ensuring the application's scalability and reliability. If you have previous experience building similar applications, please don't...

Please Sign Up or Login to see details.

We're looking to see how the latests AI rotoscoping tools look compared to one we're building ourselves. Specifically we're interested in Davinci, After Effects and Runway. We'd like someone with access to/familiarity with all of them (at least the background removal function). This is how it would work: - We give you a single clip - You use each tool to remove the background best you can while screen recording each process and making note of how many clicks it took to select the whole object - For controlled conditions, you have 30 seconds to select the foreground in each tool - Export as is (no further clean up) - Then using whatever editing tool you like the most, take these outputs (and one we'll prvide) and create identical vertical videos fo...

...Solutions: We create custom web applications that meet your specific business needs, enhancing productivity and operational efficiency. ● Versatile Applications: Our solutions range from customer portals to complex inventory management systems, each built to address unique business challenges. ● Technology Stack: Node.js, , Ruby on Rails, Django, Laravel, MongoDB, PostgreSQL, MySQL, AWS, Docker....

...government agencies in Brazil. It allows users to access multiple government platforms and services with a single set of credentials. When a user attempts to log in to a participating website or application, they are redirected to a common login page hosted at "". Here, users can enter their credentials, typically consisting of a username (usually their CPF - Cadastro de Pessoas Físicas, the Brazilian individual taxpayer registry identification) and a password. Description: I am reaching out to inquire about a service you may provide regarding the automation of login within a Docker environment. I have encountered difficulties automating the login process on , particularly when attempting to execute the script within a Docker container....

I'm in need of an experienced web developer with strong shell scripting skills. While the main purpose of the website application is not specified, it's targeted at the general public. Ideal Skills and Experience: - Proven expertise in web application development - Experience in creating websites geared towards the general public - Robust skills in shell scripting - Ability to independently make decisions to enhance website functionalities Note: As the tasks for shell scripting are undefined, a flexible way of working is preferred for meeting emergent needs in the project.

We are in search of a skilled freelancer proficient in web scraping with Scrapy to extract product data from This project involves scraping the Category page to retrieve product details including name, price, SKU, URL, breadcrumb, currency, and brand. Targeted webpage: Category page of Required data: Product name, price, SKU, URL, breadcrumb, currency, and brand Experience: Prior experience in scraping product-related information from e-commerce sites is essential, with proficiency in Scrapy being crucial for accurate and efficient data extraction. Data Handling: Ability to ensure accuracy and consistency of extracted data. Familiarity with different data formats is advantageous. Dynamic Scraping: Navigation logic for all categories

...limitations. The ideal candidate will thrive in a collaborative environment, inspire innovation, and demonstrate a self-motivated attitude toward excellence. Why Join Us: Joining our team presents an exciting opportunity to advance your career in the IT industry amid the remarkable growth of the US job market. As part of our prominent company, you'll contribute to cutting-edge projects that shape the future, positioning yourself at the forefront of innovation. With comprehensive support programs, we ensure a seamless transition into our team, empowering you to hit the ground running and excel in your role. What You Bring: - Over 8 years of experience as a Python Developer, specializing in web API development, data analytics, and machine learning. - Pro...

Hi, I have a Portainer instance running with Docker and i want to have a backup solution that backups all Containers to a external location or within the same server for me to able to restore anything once i have failures with for example updates etc.

Hi, i have been trying to update nextcloud to the latest version but was not able to. Im running it on Portainer with Docker but i dont know how to do it. We do have some customizations for logo. Also all files needs to stay on the server so we dont lose it. Require someone with experience to be able to show me how to do the update.

I'm in need of a professional who can build a tool that scrapes product data from Walmart and exports it into an Excel spreadsheet. Key Requirements: - Develop a web scraping tool specifically for Walmart website - Extract essentials (price, upc, title) Skills and Experience: - Proficiency in web scraping using Python, Scrapy, or similar tools Please let me know your availability and estimated time for completion. This is an urgent project, so a quick turnaround would be highly appreciated.

YOUR BID PRICE IS FINAL. NO LATER CHANGE IS ALLOWED. I need an expert in CWP panel and in VPS who can install Python, NodeJS websites in my server as docker container. You don't need to do any coding. My VPS server is with Hetzner and it is more than one year old. Please see the screenshot. My VPS has 4GB RAM, 80 GB Harddisk, and CentOS 7 with Root access. it is long term project. I need to install 2 Python apps as of now. My budget is INR 600 per successful installation. I will prefer AnyDesk or TeamViewer. Might need to work with my developer during installation and after installation need to clean up junk files, depreacted, not used files to save space of my harddisk. Lowest Bidder will win.

...skilled DevOps trainer who can help me in understanding the core concepts and tools of DevOps. My ideal candidate should have a concrete understanding of Docker, Jenkins, and Kubernetes. Key Requirements: - Proficient in Docker, Jenkins, and Kubernetes Additionally, I'm interested in a trainer who has hands-on experience with DevOps practices in a cloud environment. Specifically, experience with Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) would be a major plus. The training sessions should be regular and comprehensive. I'm expecting around 6-10 hours of training sessions per week. Ideal Skills and Experience: - Proficient in Docker, Jenkins, and Kubernetes - Hands-on experience with AWS...

I'm looking for an expert (8+ years) Python Django full-stack developer to become the business CO-FOUNDER. This individual must possess proficiency in back-end development and DevOps experience, database management, and server administration and must be a self-starter capable of overseeing the all website aspects. This is a position for a leader who has dreams, passion and ambition to be on the ground-floor of a business they believe in and a with the desire to want play a major role in our success. Our site and apps are in the final stages of completion and I have a team of developers currently in motion for the website and iOS/Android apps to fix bugs and optimize our operation. This responsibility would be to manage PRs, oversee performance, ta...

...extraction rates. Responsibilities: Optimize existing web scraping scripts for speed and efficiency. Deploy and manage web scraping scripts on platforms such as Scrapy Cloud, Apify, AWS Lambda, or Azure. Implement techniques to ensure high-speed data extraction (e.g., parallel processing, distributed crawling, efficient request handling). Monitor and troubleshoot scraping processes to ensure uninterrupted operation. Collaborate with our team to understand project requirements and adjust scraping strategies as needed. Requirements: Proven experience with web scraping using platforms like Scrapy Cloud, Apify, AWS Lambda, or Azure. Proficiency in Python only relevant programming commonly used for web scraping. Strong understanding of web scraping techniques, includ...

I'm looking to hire a freelancer who can develop an ACAP SDK application using Docker and the C/C++ language. Key Requirements: - Develop an ACAP SDK application - Utilize Docker for the development - Proficiency in C/C++ Language If you don't have the experience in acap and docker and the requirements please dont bid Thank you

configure haproxy with my VPS (digital ocean) - I have multiple applications - vue and django - for development Request: - configure domains and subdomains for applications - link domains wiht docker services or containers - configure Lets enctypt certificate

...that queries multiple external sources (18) through APIs, web scraping, and web page reengineering processes to extract information and structure it in the expected format. The system should generate a record of each query made, indicating whether results were obtained, no results were obtained, or if the source was not available. A document detailing all the sources of information from which data should be extracted will be provided. Main Features: 1. Automatic Querying of External Sources: Perform queries to multiple external sources according to the provided specifications. 2. Data Structuring: Process the obtained information and structure it in the required format. 3. Query Log Generation: Create a detailed log of all queries made and their results. 4. Response Handl...

Job Description: We are seeking a highly skilled Senior Software Engineer proficient in Rust, GoLang, and experienced in cloud technology. As a key member of our engineering team, you will play a crucial role in designing, developing, and deploying robust software solutions that power our platform. Responsibilities: * Collaborate with cross-functional teams to design and implement scalable, reliable, and secure software systems. * Develop high-quality code in Rust and GoLang to meet project requirements and deadlines. * Optimize software performance and ensure scalability across distributed cloud environments. * Implement best practices for software development, including code reviews, testing, and documentation. * Stay updated on emerging technolog...

I want to enlist...want to enlist the assistance of an experienced web scraping developer, specifically in scraping data from Redfin. I'm primarily interested in the following data points: - Property address - Price - The number of bedrooms - The number of bathrooms This data will be used for personal reasons therefore respectful dealings with the website and its terms of service are obligatory. Here's the kicker, I don't just need you once. I'm wanting the data to be updated on a monthly basis, so I require a solution that's reliable in the long term. Ideal skills for this job are expertise in web scraping, familiarity with real estate websites particularly Redfin, and proficiency in language like Python, BeautifulSoup, S...

I'm in urgent need of a Microsoft Azure expert who can assist with designing, deploying, and managing our Azure infrastructure. The tasks involved are crucial and require an experienced professional who can handle them with efficiency and accuracy. Key Responsibilities: - Designing Azure infrastructure: This involves creating a blueprint of the entire setup including virtual machines, storage accounts, virtual networks, and other resources. - Deploying Azure infrastructure: Once the design is finalized, the admin will be responsible for implementing the infrastructure based on the design. - Managing Azure infrastructure: After successful deployment, continuous management is required to ensure everything runs smoothly. The admin should be able to troubleshoot issues, optimiz...

Looking for a skilled developer to assist with configuring Apache Kafka for a distributed messaging system. I have running zookeeper, kafka-server and kafka-connect. the p...kafka-server and kafka-connect. the project requires: 1) configure kafka source connector for receiving a Json and store it on a topic. 2) configure a kafka sink connector for storing the json data into a PostgreSQL table 3) guidance for installing the plugins 4) setup the connectors over kafka-api the kafka server will be running on ubunut 24.x without a docker Ideal Skills and Experience: - Familiarity with Apache Kafka platform - Experience in distributed messaging - Strong problem-solving skills - Attention to detail We are eager to work with someone who is able to provide solutions promptly...

This role is responsible for the development, deployment, and management of a decentralized AI and GPU-sharing platform. The ideal candidate will combine expertise in AI, blockchain technology, and systems infrastructure to drive the implementation and scalability of the platform. Responsibilities: Design and develop a Layer 1 blockchain infrastructure specifically optimized for task distribution in decentralized environments. Design, implement, and maintain AI services using popular frameworks such as TensorFlow, PyTorch, Caffe2, and MXNet. Develop blockchain components, focusing on smart contracts and decentralized applications (DApps) integration. Manage cloud-based and on-premise GPU nodes, including deployment, scaling, and monitoring. Ensure system reliability and sca...

I'm looking to set up a Debian system, install Docker, and then Home Assistant for me. - The initial task involves configuring a Debian-based machine. This will require expertise in Debian, including experience with the system's configurations and best practices. - The next step would involve setting up Docker. My intention is to use containers for various purposes. I'd appreciate if you could advise on the most suitable options, given that the specific use cases for the containers have not been provided. - Lastly, you will install and configure Home Assistant. My main goal for this is to have home automation, security monitoring, and energy management capabilities. - The devices I plan to integrate with Home Assistant include smart lights, smart ...

You can got through my freelancer.com profile. I am a fulltime freelancer with 5 star rating and awesome reviews. Profile links https://www.freelancer.com/u/Puneet84370?sb=t These are my details Name - PUNEET BAJAJ ACHIEVEMENTS JEE Main 2023 : Achieved a 99.29 percentile, reflecting dedication and effort. JEE Advanced 2023 : Among top 11k students out of 1.9 lac test takers NAEST 2022 : Among only 1000 students selected out of 1.15 lakh from 8th to M.Sc. IOQM 2022 : Qualified Part A of toughest Math exam for students & wrote INMO . Harvard Ready Player 50 : Among only 3700 across the world to solve all 7 levels . CERTIFICATIONS: Google Advanced Data Analytics Professional Certificate | Google - View Credential

Please Sign Up or Login to see details.

...permissions. Specifically, when it comes to executing certain commands, I'm greeted with a 'Permission denied' error. This is particularly impacting my ability to make set the download folder for sabnzbd. This folder is a remote SMB share mounted to a ubuntu VM. sabnzbd is running inside a docker container. Key Requirements: - Resolve the 'Permission denied' issue for commands execution - Enable editing and modifying of sabnzbd - Ensure the affected permissions are tailored for a single user Your experience and skills in Linux systems and permissions are crucial for this project. An understanding of both installed software application permissions and how to adjust them for a single user will be beneficial. Your ability to troubleshoot and resolv...

...need an expert with solid experience with AWS services, particularly Elastic Beanstalk and S3. As part of this project, I require: - Debug two Docker applications - Deployment of two applications on Elastic Beanstalk within my existing architecture. - Set up of two S3 buckets intended to store app data and archive data. The data to be stored is expected to be in the medium range (1-10 GB). Therefore, the buckets must be optimized for this scale. Required skills and experiences: - AWS Elastic Beanstalk: It's crucial for you to understand and proficiently manage application deployment within my infrastructure. - AWS S3: You need to have expertise in creating and managing S3 buckets, specifically for storing application data and archiving purposes. The mai...

I'm programming a sensitive app where I don't want to have full control over the deployment process / cloud account. Therefore I need a reviewer / ops person to take ownership of the deployment process. I need an hour or so of your time every week to: - Review a pull request created by me - Run a deployment pipeline - Confirm project has deployed successfully - Review cloud account and indicate any issues (e.g. unpaid invoice, server downtime, etc.) Estimated effort / time required: 1-2hrs per week. Note: You don't need to be a programmer / specialist. Just technically literate.

I need someone to assist me in setting up the Zabor POS APK on my Windows 10 computer. I would like the APK to run on BlueStacks, an Android emulator. This is for personal use, so it is important that the setup is seamless and runs smoothly. Key Responsibilities: - Install and configure BlueStacks for optimal performance - Download, install, and set up the Zabor POS APK - Ensure the APK is running smoothly on Windows 10 Ideal Skills and Experience: - Strong knowledge of Android emulators, particularly BlueStacks - Experience in setting up and configuring software - Attention to detail to ensure smooth operation Please note that I require assistance with the initial setup and configuration - not just running the APK. Your help is appreciated.

This is a simple C++ command line program with parameters. I don't work with C++ on a daily basis and I need support. The program uses an external SDK, and I have header, lib and dll files. It needs to be connected and configured for everything to work. I present screenshots. This needs to be configured on my computer, I can share the screen remotely.

I'm looking for a skilled React.js developer to help me run the app locally using or deploy it to sever using anydesk. I will provide a great detailed review. @danielhvoigt/how-to-make-your-react-app-open-locally-through-an-index-html-file-413b0405907

I am in need of an expert who can give assistance with installing OpenDevin on my windows platform. I try to follow this tutorial But I can't I have a problem with docker could not be found in this WSL2distro Project to install on Win 10 64bits

...requiring a data scrapping software quickly to extract text content from both websites and PDF documents. It's crucial that the scraped data is directed into Excel or Google Sheets format for my immediate use. Ideal skills for this job would be: - Proficiency in programming languages like Python that are often used for data scrapping. - Familiarity with web scraping libraries or frameworks, e.g., Beautiful Soup or Scrapy. - Experience using APIs to scrape data if necessary. - Previous work in pulling data from PDFs into readable formats in Excel or Google Sheet. - Ability to deliver the project ASAP. Your attention to these requirements will be much appreciated as I need this task done urgently. Please contact me if you need more detail about the proj...