DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
You will also need docker installed on your system. I recommend docker-ce on Ubuntu 1622.04. See the Jenkins bootstrap script or the Docker documentation for steps to install docker-ce.
...
| Code Block | ||
|---|---|---|
| ||
./buildall.sh -release -skiptests -ninja -noclean -notests
ninja docker_images |
| Info |
|---|
You may want to set USE_CDP_HIVE in bin/impala-config-local.sh to build containers with Hive 3 support |
| Info |
|---|
Note - in future it would be nice if we could have a flag to buildall.sh to also build this target. |
...
| Code Block |
|---|
$ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE impala_base latest ffb1347a05b7 2 weeks ago 2.91GB impalad_coord_exec latest dbea9fb26675 4 weeks ago 2.39GB impalad_coordinator latest bd213630acce 4 weeks ago 2.39GB statestored latest 59372e24a8f7 4 weeks ago 2.39GB catalogd latest d56a5adf86d0 4 weeks ago 2.39GB impalad_executor latest a735455d0c83 4 weeks ago 2.39GB |
Pushing Images to a Repository
./docker/push-images.sh is a script that can push the built images to a docker repository. See that script's help for more information.
Running Dockerized Minicluster
...
| Code Block |
|---|
start-impala-cluster.py --kill
start-impala-cluster.py --docker_network=impala-cluster-network
impala-shell.sh
|
Note that querying any existing tables is likely to fail because "localhost" is baked into a lot of metadata. You will need to load, or re-load data before running end-to-end tests.
You can see the running docker containers with "docker ps"
| Code Block |
|---|
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
af295db2f818 impalad_coord_exec "/opt/impala/bin/dae…" 2 minutes ago Up 2 minutes 0.0.0.0:21002->21000/tcp, 0.0.0.0:21052->21050/tcp, 0.0.0.0:25002->25000/tcp impala-test-cluster-impala-cluster-network-impalad_coord_exec-2
57011eda4ded impalad_coord_exec "/opt/impala/bin/dae…" 2 minutes ago Up 2 minutes 0.0.0.0:21001->21000/tcp, 0.0.0.0:21051->21050/tcp, 0.0.0.0:25001->25000/tcp impala-test-cluster-impala-cluster-network-impalad_coord_exec-1
7a3caac8ccc0 impalad_coord_exec "/opt/impala/bin/dae…" 2 minutes ago Up 2 minutes 0.0.0.0:21000->21000/tcp, 0.0.0.0:21050->21050/tcp, 0.0.0.0:25000->25000/tcp impala-test-cluster-impala-cluster-network-impalad_coord_exec-0
e54887a3bc2c catalogd "/opt/impala/bin/dae…" 2 minutes ago Up 2 minutes 0.0.0.0:25020->25020/tcp impala-test-cluster-impala-cluster-network-catalogd
57cd35033b15 statestored "/opt/impala/bin/dae…" 2 minutes ago Up 2 minutes 0.0.0.0:25010->25010/tcp impala-test-cluster-impala-cluster-network-statestored |
Running Tests
Automated end-to-end tests are run with this job: https://jenkins.impala.io/job/ubuntu-1620.04-dockerised-tests/. https://github.com/apache/impala/blob/master/bin/jenkins/dockerized-impala-bootstrap-and-test.sh bootstraps ubuntu16.04 ubuntu from scratch and runs the tests.
...
- The Impala debug pages are mostly exposed on the same ports as the regular minicluster, i.e. localhost:25000, localhost:25001, localhost:25002, localhost:25010, localhost:25020. This is achieved by mapping from the default webserver ports inside the container to the desired ports outside the container. I.e. all of the Impala daemons are exposing their webserver on port 25000 inside the container.
- If you want to look at logs or other state in a running container, you can use "docker exec" to run a bash process inside a container (using the name or ID from "docker ps"). Inside the container is a very stripped down Ubuntu environment, so it may be missing many commands you're accustomed to!
| Code Block |
|---|
docker exec -it impala-test-cluster-impala-cluster-network-catalogd /bin/bash
cat /tmp/catalogd.INFO |
...