<p>I have 3 microservices, where service A talks to service B and C, and C can talk to A.</p>
<p>I have set up a Jenkins pipeline, where unit tests get run as part of the CI build, however, I'm not sure how to test the container once deployed on the pod. Integration tests on our staging server? I have looked into creating a test container, and using that as a test runner, which can populate the database with data, and verify the results, and produce a report. Is this how it's done?</p>

user1555190
<h2>Answer</h2>
<p>Yes, the approach described in your question is an effective way of testing distributed services on Kubernetes.</p>
<p>We maintain a staging cluster that CI pipelines push microservices to and perform tests on.</p>
<p>Each pipeline will run unit tests and then build a Docker image, tag it, and push it to our registry. After this, the pipeline will run something like <code>kubectl set image ...</code> to initiate a rollout of a new ReplicaSet for the respective microservice. Note that we use a unique tag for the image (not latest, for example).</p>
<p>We wait for the rollout via <code>kubectl rollout status ...</code> and once the rollout is complete, the pipeline begins running tests.</p>
<p>We usually run feature and integration tests directly from the pipeline VM. We run load tests (e.g. using locust.io) in a Pod deployed on the staging cluster.</p>
<p>If the tests fail, we run <code>kubectl rollout undo ...</code> to revert the microservice to the previous image.</p>
<h2>More Info</h2>
<p>Its good practice to maintain Deployment and Service YAML spec in each microservice project (e.g. in deploy/ dir) so that the CI pipeline can be idempotent. In this case you would run <code>kubectl apply -f deploy/*.yaml</code> to update your microservices on the staging cluster.</p>
<p>Pods running test fixtures can rely on ClusterIP services when running against your on-cluster microservices. Tests running from pipeline VMs will need to hit your microservices through Services that have an external IP.</p>
