Docker For Mac Slow Ext4 Partition

| 25-03-2021
[RAND-1-100] Comments

Mar 13, 2020 September 2020 Update: Alas, Docker for Mac will not be getting built-in Mutagen support at this time. Ever since Docker for Mac was released, shared volume performance has been a major pain point. It was painfully slow, and the community finally got a cached mode that offered a 20-30x speedup for common disk access patterns around.

/ has Docker images (via /var/lib/docker) which is the concern: docker can fill the root partition. /srv has Jenkins build results (large), zuul-merge repositories ( 25GB), integration.wikimedia.org docroot (small) We would want to shrink the volume group and create a new one for Docker images which would be at /srv/docker. Aug 07, 2017 CentOS 7 though creates EXT4 filesystems with dtype=1 enabled by default. See explanation at What is dtype and why Docker overlayfs need it - Linuxer Pro. Worse thing is you can't fix it on existing CentOS 7 created XFS partitions. You need to recreate the filesystem if you setup CentOS 7 filesystem with XFS partitions! Oct 14, 2019 A couple of years ago, Will Pleasant-Ryan wrote Docker for Mac: Overcoming Slow Mounted Volumes, describing his desire to use Docker for local development. He talked about some of the filesystem performance problems that can arise when using a shared volume from the host machine on Mac OS X, along with some potential workarounds and word of some upcoming performance improvements.

Estimated reading time: 16 minutes

Welcome to Docker Desktop! The Docker Desktop for Mac user manual provides information on how to configure and manage your Docker Desktop settings.

For information about Docker Desktop download, system requirements, and installation instructions, see Install Docker Desktop.

Preferences

The Docker Preferences menu allows you to configure your Docker settings such as installation, updates, version channels, Docker Hub login,and more.

Choose the Docker menu > Preferences from themenu bar and configure the runtime options described below.

General

On the General tab, you can configure when to start and update Docker:

  • Start Docker Desktop when you log in: Automatically starts Docker Desktop when you open your session.

  • Include VM in Time Machine backups: Select this option to back up the Docker Desktop virtual machine. This option is disabled by default.

  • Securely store Docker logins in macOS keychain: Docker Desktop stores your Docker login credentials in macOS keychain by default.

  • Send usage statistics: Docker Desktop sends diagnostics, crash reports, and usage data. This information helps Docker improve and troubleshoot the application. Clear the check box to opt out.

Resources

The Resources tab allows you to configure CPU, memory, disk, proxies, network, and other resources.

Advanced

On the Advanced tab, you can limit resources available to Docker.

Advanced settings are:

CPUs: By default, Docker Desktop is set to use half the number of processorsavailable on the host machine. To increase processing power, set this to ahigher number; to decrease, lower the number.

Memory: By default, Docker Desktop is set to use 2 GB runtime memory,allocated from the total available memory on your Mac. To increase the RAM, set this to a higher number. To decrease it, lower the number.

Swap: Configure swap file size as needed. The default is 1 GB.

Disk image size: Specify the size of the disk image.

Disk image location: Specify the location of the Linux volume where containers and images are stored.

You can also move the disk image to a different location. If you attempt to move a disk image to a location that already has one, you get a prompt asking if you want to use the existing image or replace it.

File sharing

Use File sharing to allow local directories on the Mac to be shared with Linux containers.This is especially useful forediting source code in an IDE on the host while running and testing the code in a container.By default the /Users, /Volume, /private, /tmp and /var/folders directory are shared. If your project is outside this directory then it must be addedto the list. Otherwise you may get Mounts denied or cannot start service errors at runtime.

File share settings are:

  • Add a Directory: Click + and navigate to the directory you want to add.

  • Apply & Restart makes the directory available to containers using Docker’sbind mount (-v) feature.

Tips on shared folders, permissions, and volume mounts

  • Shared folders are designed to allow application code to be edited on the host while being executed in containers. For non-code items such as cache directories or databases, the performance will be much better if they are stored in the Linux VM, using a data volume (named volume) or data container.

  • By default, Mac file systems are case-insensitive while Linux is case-sensitive. On Linux, it is possible to create 2 separate files: test and Test, while on Mac these filenames would actually refer to the same underlying file. This can lead to problems where an app works correctly on a Mac (where the file contents are shared) but fails when run in Linux in production (where the file contents are distinct). To avoid this, Docker Desktop insists that all shared files are accessed as their original case. Therefore, if a file is created called test, it must be opened as test. Attempts to open Test will fail with the error No such file or directory. Similarly, once a file called test is created, attempts to create a second file called Test will fail. For more information, see Volume mounting requires file sharing for any project directories outside of /Users.)

Proxies

Docker Desktop detects HTTP/HTTPS Proxy Settings from macOS and automaticallypropagates these to Docker. For example, if you set yourproxy settings to http://proxy.example.com, Docker uses this proxy whenpulling containers.

Your proxy settings, however, will not be propagated into the containers you start.If you wish to set the proxy settings for your containers, you need to defineenvironment variables for them, just like you would do on Linux, for example:

For more information on setting environment variables for running containers,see Set environment variables.

Network

You can configure Docker Desktop networking to work on a virtual private network (VPN). Specify a network address translation (NAT) prefix and subnet mask to enable Internet connectivity.

Docker Engine

The Docker Engine page allows you to configure the Docker daemon to determine how your containers run.

Type a JSON configuration file in the box to configure the daemon settings. For a full list of options, see the Docker Enginedockerd commandline reference.

Click Apply & Restart to save your settings and restart Docker Desktop.

Command Line

On the Command Line page, you can specify whether or not to enable experimental features.

Experimental features provide early access to future product functionality.These features are intended for testing and feedback only as they may changebetween releases without warning or can be removed entirely from a futurerelease. Experimental features must not be used in production environments.Docker does not offer support for experimental features.

To enable experimental features in the Docker CLI, edit the config.jsonfile and set experimental to enabled.

To enable experimental features from the Docker Desktop menu, clickSettings (Preferences on macOS) > Command Line and then turn onthe Enable experimental features toggle. Click Apply & Restart.

For a list of current experimental features in the Docker CLI, see Docker CLI Experimental features.

You can toggle the experimental features on and off in Docker Desktop. If you toggle the experimental features off, Docker Desktop uses the current generally available release of Docker Engine.

You can see whether you are running experimental mode at the command line. IfExperimental is true, then Docker is running in experimental mode, as shownhere. (If false, Experimental mode is off.)

Kubernetes

Docker Desktop includes a standalone Kubernetes server that runs on your Mac, sothat you can test deploying your Docker workloads on Kubernetes.

The Kubernetes client command, kubectl, is included and configured to connectto the local Kubernetes server. If you have kubectl already installed andpointing to some other environment, such as minikube or a GKE cluster, be sureto change context so that kubectl is pointing to docker-desktop:

If you installed kubectl with Homebrew, or by some other method, andexperience conflicts, remove /usr/local/bin/kubectl.

  • To enable Kubernetes support and install a standalone instance of Kubernetesrunning as a Docker container, select Enable Kubernetes. To set Kubernetes as thedefault orchestrator, select Deploy Docker Stacks to Kubernetes by default.

    Click Apply & Restart to save the settings. This instantiates images required to run the Kubernetes server as containers, and installs the/usr/local/bin/kubectl command on your Mac.

    When Kubernetes is enabled and running, an additional status bar item displaysat the bottom right of the Docker Desktop Settings dialog.

    The status of Kubernetes shows in the Docker menu and the context points todocker-desktop.

  • By default, Kubernetes containers are hidden from commands like dockerservice ls, because managing them manually is not supported. To make themvisible, select Show system containers (advanced) and click Apply andRestart. Most users do not need this option.

  • To disable Kubernetes support at any time, clear the Enable Kubernetes check box. TheKubernetes containers are stopped and removed, and the/usr/local/bin/kubectl command is removed.

    For more about using the Kubernetes integration with Docker Desktop, seeDeploy on Kubernetes.

Reset

Reset and Restart options

On Docker Desktop Mac, the Restart Docker Desktop, Reset to factory defaults, and other reset options are available from the Troubleshoot menu.

For information about the reset options, see Logs and Troubleshooting.

Dashboard

The Docker Desktop Dashboard enables you to interact with containers and applications and manage the lifecycle of your applications directly from your machine. The Dashboard UI shows all running, stopped, and started containers with their state. It provides an intuitive interface to perform common actions to inspect and manage containers and existing Docker Compose applications. For more information, see Docker Desktop Dashboard.

Add TLS certificates

You can add trusted Certificate Authorities (CAs) (used to verify registryserver certificates) and client certificates (used to authenticate toregistries) to your Docker daemon.

Add custom CA certificates (server side)

All trusted CAs (root or intermediate) are supported. Docker Desktop creates acertificate bundle of all user-trusted CAs based on the Mac Keychain, andappends it to Moby trusted certificates. So if an enterprise SSL certificate istrusted by the user on the host, it is trusted by Docker Desktop.

To manually add a custom, self-signed certificate, start by adding thecertificate to the macOS keychain, which is picked up by Docker Desktop. Here isan example:

Or, if you prefer to add the certificate to your own local keychain only (ratherthan for all users), run this command instead:

See also, Directory structures forcertificates.

Note: You need to restart Docker Desktop after making any changes to thekeychain or to the ~/.docker/certs.d directory in order for the changes totake effect.

For a complete explanation of how to do this, see the blog post AddingSelf-signed Registry Certs to Docker & Docker Desktop forMac.

Add client certificates

You can put your client certificates in~/.docker/certs.d/<MyRegistry>:<Port>/client.cert and~/.docker/certs.d/<MyRegistry>:<Port>/client.key.

When the Docker Desktop application starts, it copies the ~/.docker/certs.dfolder on your Mac to the /etc/docker/certs.d directory on Moby (the DockerDesktop xhyve virtual machine).

  • You need to restart Docker Desktop after making any changes to the keychainor to the ~/.docker/certs.d directory in order for the changes to takeeffect.

  • The registry cannot be listed as an insecure registry (see DockerEngine. Docker Desktop ignores certificates listedunder insecure registries, and does not send client certificates. Commandslike docker run that attempt to pull from the registry produce errormessages on the command line, as well as on the registry.

Directory structures for certificates

If you have this directory structure, you do not need to manually add the CAcertificate to your Mac OS system login:

The following further illustrates and explains a configuration with customcertificates:

You can also have this directory structure, as long as the CA certificate isalso in your keychain.

To learn more about how to install a CA root certificate for the registry andhow to set the client TLS certificate for verification, seeVerify repository client with certificatesin the Docker Engine topics.

Install shell completion

Ext4

Docker Desktop comes with scripts to enable completion for the docker and docker-compose commands. The completion scripts may befound inside Docker.app, in the Contents/Resources/etc/ directory and can beinstalled both in Bash and Zsh.

Bash

Bash has built-in support forcompletion To activate completion for Docker commands, these files need to becopied or symlinked to your bash_completion.d/ directory. For example, if youinstalled bash via Homebrew:

Add the following to your ~/.bash_profile:

OR

Zsh

In Zsh, the completionsystemtakes care of things. To activate completion for Docker commands,these files need to be copied or symlinked to your Zsh site-functions/directory. For example, if you installed Zsh via Homebrew:

Fish-Shell

Fish-shell also supports tab completion completionsystem. To activate completion for Docker commands,these files need to be copied or symlinked to your Fish-shell completions/directory.

Create the completions directory:

Now add fish completions from docker.

Give feedback and get help

To get help from the community, review current user topics, join or start adiscussion, log on to our Docker Desktop for Macforum.

To report bugs or problems, log on to Docker Desktop for Mac issues onGitHub,where you can review community reported issues, and file new ones. SeeLogs and Troubleshooting for more details.

For information about providing feedback on the documentation or update it yourself, see Contribute to documentation.

Docker Hub

Select Sign in /Create Docker ID from the Docker Desktop menu to access your Docker Hub account. Once logged in, you can access your Docker Hub repositories and organizations directly from the Docker Desktop menu.

For more information, refer to the following Docker Hub topics:

Two-factor authentication

Docker Desktop enables you to sign into Docker Hub using two-factor authentication. Two-factor authentication provides an extra layer of security when accessing your Docker Hub account.

You must enable two-factor authentication in Docker Hub before signing into your Docker Hub account through Docker Desktop. For instructions, see Enable two-factor authentication for Docker Hub.

After you have enabled two-factor authentication:

  1. Go to the Docker Desktop menu and then select Sign in / Create Docker ID.

  2. Enter your Docker ID and password and click Sign in.

  3. After you have successfully signed in, Docker Desktop prompts you to enter the authentication code. Enter the six-digit code from your phone and then click Verify.

After you have successfully authenticated, you can access your organizations and repositories directly from the Docker Desktop menu.

Where to go next

  • Try out the walkthrough at Get Started.

  • Dig in deeper with Docker Labs examplewalkthroughs and source code.

  • For a summary of Docker command line interface (CLI) commands, seeDocker CLI Reference Guide.

  • Check out the blog post, What’s New in Docker 17.06 Community Edition(CE).

Read Ext4 Mac

mac, tutorial, run, docker, local, machine

View Ext4 Partition Windows

A couple of years ago, Will Pleasant-Ryan wrote Docker for Mac: Overcoming Slow Mounted Volumes, describing his desire to use Docker for local development. He talked about some of the filesystem performance problems that can arise when using a shared volume from the host machine on Mac OS X, along with some potential workarounds and word of some upcoming performance improvements.

In the time since that post was written, those improvements have been released. But I’ve still found it to be unacceptably slow to have the source code mounted from the host machine, at least for something like a Ruby on Rails application.

However, earlier this summer, a Visual Studio Code update introduced the ability to “attach to a running container.” Once attached, the experience when editing files that reside on a container using VS Code is nearly indistinguishable from that of editing files on the local filesystem with VS Code!

1. Docker Compose

I’m using Docker Compose to manage the containers that make up my development environment. There’s plenty of existing documentation on how to use Docker Compose, but I want to point out a couple of things that I’ve done because I’m using a container to host the source code I’m editing.

A named volume for source code

If you just checked out a repository into a normal directory inside the container, you’d have to be careful not to lose any changes that had not been committed and pushed whenever making changes to container (updating something in the Dockerfile, etc.). But if the source code is checked out into a named volume, the container can be re-built over and over again, with your source code being mounted exactly as it was before.

A long-running container

Similar to a database container, your “dev” container should stay running when you do a docker-compose up so you can attach to it from VS Code. With Docker Compose, you just need to set the tty option to true in your docker-compose.yml.

Here’s an example docker-compose.yml file that has a couple of named volumes, one for the source code directory and the other for the contents of the database. I’m using this for an old Ruby on Rails project, hence the old version of MySQL.

2. Working in the Dev Container

The idea is that any developer could check out the repository on their laptop, run docker-compose up, and have a working development environment. But you’ll still need a little customization of the container once it’s been built.

Check out the source code

Once you have your container up and running (docker-compose up), you’ll need to check out your source code inside the container. You can’t do this as part of the Dockerfile because we want the repository to live in the mounted volume.

If you rely on your SSH keys to authenticate with your Git server, then you’ll want to either mount your ~/.ssh directory in the container or copy your private key into the container manually. I needed to do the latter because my ~/.ssh/config has some things that aren’t compatible with the Ubuntu version running in the container.

These examples assume the container will be run as a user called “dev.”

I also like to have my normal Git configuration available as well:

Now you just check out your source code, but inside the container instead of on your host machine.

At this point, you can open a shell in the container and run tests, start the development server, etc.

3. Editing in VS Code

With the “dev” container running, inside of Visual Studio Code, run the “Remote-Containers: Attach to Running Container” command and select your “dev” container. VS Code will attach and install everything it needs to run on the container.

If you select the File Explorer, you’ll see an “Open Folder” button that, when clicked, will display folders on the container. Select the location of the source code that was checked out previously, and click OK. You’ll now be able to edit files as if they were hosted on the local filesystem.

You can even open an integrated terminal in VS Code, and it will automatically open a shell in the attached container!

Conclusion

I’ve only recently started playing with hosting my entire development environment, including the source code, inside Docker containers, but the experience using VS Code as an editor has been fantastic.

With the Mac OS X filesystem integration performance improvements that have been made, many projects will be just fine using shared volumes mounted from the host system. But in cases where that just won’t cut it, hosting the source in the container and using VS Code to attach to the container is a really, really nice solution.