Is it possible to run dotnet-docker based containers with memory limits?

[geoffpatehutch]

I'm trying to use a dotnet-docker based container in a limited memory environment. I am passing --memory 134217728 (128MB) and --memory-swap=0 on the docker command line via DC/OS running on Ubuntu in the Azure Container Service.

The problem I have is that the .NET garbage collector is not doing what I expect. I have a simple timer that reloads data from a database creating small amounts of garbage regularly. I can see the memory usage increase until the container is killed due to OOM constraints. I would have expected the garbage collector to free up memory before this happened.

I am wondering if the container believes it has access to as much memory as the container host (3.5GB) and therefore isn't scheduling garbage collection as it feels under no pressure?

[richlander]

Good question. We have talked about this and I can see why it would be a good idea. We will look into it. I expect we have some tests that will re-create this situation so that we can validate it ourselves.

/cc @Maoni0 @MichaelSimons @glennc @kendrahavens

[geoffpatehutch]

Just a little more info - I ran the same container with increased memory allocation (2GB), and the memory climbed steadily until it hit 160MB. At that point there was a big drop back down to ~86-88MB and it has stayed there since. Not sure why 160MB would be a magic number, but thought it might be of some use in your investigations!

[MichaelSimons]

FWIW I created a simple .NET Core console application that continuously allocates memory (arrays of string). As I run it in a container and monitor it with docker stats it seems to behave as I suspect. By that I mean if I hold onto the references, the memory usage will continue to grow until it hits the hard limit set by --memory (w/--memory-swap=0) and will automatically get killed by Docker. If I set --memory-swap=-1, the program will continue running. Also if I release references, I can see GC kicking in and freeing up memory.

[Maoni0]

handling limited memory situation on Linux is something that @rahku is currently working on. however, could we please first make sure it is the GC that would help in this case, ie, it's memory on the GC heap that keeps growing, not due to something else. You could either induce a GC yourself, or use sos to look at the GC heap. If it's indeed the GC heap that seems to not be aware of the limit you specified, then we can certainly try out @rahku's changes.

[rahku]

It will be super helpful if you could try changes in PR dotnet/coreclr#10064 and provide feedback if this works as you would expect it to behave.

[geoffpatehutch]

I tried adding a call to GC.Collect and the problem disappeared which led me to believe it was a GC problem. I'm happy to try @rahku 's change - how would I go about doing that? At the moment I build my container using "FROM microsoft/dotnet:1.1-runtime". Is there an alternative image base somewhere I can use to incorporate these changes?

[Maoni0]

@rahku could you please help @geoffpatehutch with getting a current build to try (or whatever that's easier for him to invoke your changes)?

[rahku]

@geoffpatehutch the changes are in .net core 2.0 which is not released yet. Easiest way to try this would be to wait for the PR to merge and then you can get setup files to install from core-setup repo. I should be able to merge this early next week.

[MichaelSimons]

@geoffpatehutch, Once this gets checked in and picked up by CLI, you can use the microsoft/dotnet-nightly:2.0-sdk image from the dotnet-nightly repo to verify the change.

[rahku]

@geoffpatehutch could you try the nightly container images. It has my changes now.

[MichaelSimons]

@geoffpatehutch - did you ever get a change to try this? 2.0 preview images are now also available on microsoft/dotnet.

[geoffpatehutch]

I haven't yet as its not quite as simple as tweaking the dockerfile to use the dotnet 2.0 runtime, I need to set some time aside to install the new version and compile with it. I assume you can install 1.1 and 2.0 side by side and just switch targets at compile time?

[axelbodo]

there is another issue and a referenced link in github: moby/moby#20688
For our java applications we can temporarily solve the issues by injecting cgroup limit as a variable to use as max heapsize at java commandline. unfortunatelly dotnetcoe doesn't have such an option so their decisions is still based on what is reported by /proc and/or sysinfo, and they reports host resources, not cgroup limits. I think the temporary solution would be if dotnet(run) also would have an option for setting max heap size. The final solution which is under wide discussion is that the docker container reports container limits, not host values via /proc and sysinfo.

[rahku]

dotnet core is still based on what is reported by /proc and/or sysinfo

@axelbodo this is not correct for .net core 2.0. .net core 2.0 reads memory limits for cgroups if they are set.

[axelbodo]

Thanks for the info, we use now netcore 1.1, and I don't know when we are enabled to go to 2.0. Am I right it has only a preview release now?