Skip to content

GET certain paths like systemVM and networkofferings are very very slow #12740

@Sparc0

Description

@Sparc0

problem

Background
We have just recently deployed our first production Cloudstack environment.
At the moment we have 2 zones 1 pod 1 cluster, with 4 hosts each inside.
In total we have created 4 extra networkofferings over the default ones, 5 shared networks and 5 instances.
And we have 4 systemvm's.
So as you can understand its still a very small deployment still.

Our management server is a VM.
Running on a oVirt cluster.
VM is 4 cores and 8GB ram.
MariaDB and CS Mangement server are located on the same VM.

Issue
When we try to GET certain paths either if its using the webui or cloudmonkey script the load times are very long and our MariaDB goes to 100% for a while on one core.
Path where the issue is seen are:
GET systemVMs ~45 seconds
GET networkofferings ~50seconds

Image

Doing following select queries towards the MariaDB the response back is instant and no load seen on the DB.

select * from network_offerings; 
select * from console_proxy;
select * from vm_instance;
select * from secondary_storage_vm;

More Background
During the setup we accidently imported each ldap user entry as a domain resulted in ~52k now inactive domains. I believe that we started seeing this issue after that was done. We have deleted each domain but they still show up when doing a select * from domain but as inactive.
I also remember that we had alot of systemvm entries during the first cluster because cloudstack had some network issue and it created alot of systemvm entries. But they have also been deleted.

I have enabled expunged.resources.purge.enabled
And i have cleaned up our event and alert tables.

versions

ACS 4.22

The steps to reproduce the bug

Not been able to reproduce the issue.

What to do about it?

No response

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Status

    Todo

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions