Linux Connecton Refused Issue

[BVeeravelu]

Hi,

Problem Report: Plastic SCM version 7 installed in Linux often hungs and throws error "No connection could be made becasue the target machine actively refused it <IP:PORT>". 
Action Taken/Workaround: 
Checked the status of Plastic SCM service and found it's active. 
Tried stop and start the service but couldn't because it throws error saying operation FAILED.
Reboot the Linux server, then able to establish connection.
Note: We have two other Plastic SCM running in two different Windows machine without any issues for years. But the Linux one started few months ago had connection issue reported once a week. This issue mostly occurs during 12 to 4 AM EST (onshore off business hours, but Offshore's active check-in hours)

Objective: Plastic SCM should be available and able to connect all time without any connection issue, other than maintenance downtime.

Question(s):

1. Any input and thoughts on What's causing the connection issue?
2. Please let me know for any log files required to take a look.

Please let me know for any further information or question.

Thanks,
Babu

[BVeeravelu]

Hi,

Attaching the log files for reference.

Thanks,

Babuplastic.relevant.log.txtplastic.realtime.requests.log.txtplastic.errors.log.txtplastic.debug.log.txtplastic.channelcall.log.txt

[calbzam]

Hi,

Checking your logs, I cannot see any logged info between 00:35 and 05:24 (the time you restarted the service). This log could fit with a situation where the Plastic server daemon was killed, the machine stopped...

How did you check that status of the Plastic server was active?

2019-04-25 00:35:04,584 W-21 00000000-0000-0000-0000-000000000000 S-1-5-21-625948072-1123881165-1540833222-87283 CXVM-OC63 DEBUG LdapProvider - LdapSearchResult processed in 2 ms
2019-04-25 00:35:04,584 W-21 00000000-0000-0000-0000-000000000000 S-1-5-21-625948072-1123881165-1540833222-87283 CXVM-OC63 DEBUG LdapProvider - Search finished in 2 ms
2019-04-25 00:35:04,584 W-21 00000000-0000-0000-0000-000000000000 S-1-5-21-625948072-1123881165-1540833222-87283 CXVM-OC63 INFO  LdapProvider - <- GetADSEIDGroups in 2 ms
2019-04-25 05:24:23,685 Main-1    INFO  Daemon - Initialize Datalayer
2019-04-25 05:24:24,152 Main-1    INFO  DatalayerFs.FsConfParser - JET SETTINGS:
2019-04-25 05:24:24,152 Main-1    INFO  DatalayerFs.FsConfParser - basepath=/data01

Regards,

Carlos.
 

[BVeeravelu]

Hi,

Thanks for the quick response, 

For today's issue, I couldn't check the status because I was not called. But for the issue reported on last Saturday April 20, 2019. I did try the below command after logging into the server.

$ service plasticsd status

Also, when I tried restart the service by entering first 

$ service plasticsd stop 

command, it threw a "FAILED" message, and then when I tried for 

$ service plasticsd start

it resulted in saying service already running.

I will try to get the logs for the day of April 20, 2019 to see if that helps.

[BVeeravelu]

Hi,

Attached the log files of April 19 & 20, 2019 for your reference.

Thanks,

plastic.relevant.log.txt.20190420plastic.relevant.log.txt.20190419plastic.errors.log.txt.20190420plastic.errors.log.txt.20190419plastic.debug.log.txt.20190420plastic.debug.log.txt.20190419plastic.channelcall.log.txt.20190420Babuplastic.channelcall.log.txt.20190419

[calbzam]

Hi,

The server was getting out of threads and was not responding:

2019-04-19 21:54:03,690 ActivityScheduler-20    INFO  ServerStats - ACTIVE METHOD CALLS
2019-04-19 21:54:03,690 ActivityScheduler-20    INFO  ServerStats - Name                                     IP              Time      Thread   User
2019-04-19 21:54:03,690 ActivityScheduler-20    INFO  ServerStats - GetSeid                                  10.169.51.54    00:00:00  22        
2019-04-19 21:54:03,691 ActivityScheduler-20    INFO  ServerStats - GetSeid                                  10.169.51.54    00:00:00  23        
2019-04-19 21:54:03,691 ActivityScheduler-20    INFO  ServerStats - Total calls: 2
2019-04-19 21:54:03,691 ActivityScheduler-20    INFO  ServerStats - 2 requests enqueued. 2 worker threads (0 free)

Maybe a temporary issue when connecting to the AD to resolve the SEID?

How many cores has your server? I would increase the thread number from the web admin panel:

I would also upgrade the server version to the last 8.0.

Regards,

Carlos.

[BVeeravelu]

Hi Carlos,

Thanks for the input.

Your question & Suggestion: How many cores has your server?I would increase the thread number from the web admin panel:

My answer and action taken on suggestion: We had 2 cores and 2 threads, as per your recommendation we had increased to 4 cores and matched with 4 threads now.

Question to you: You above screen shot shows 4 circled but max threads as 16, Could you please elaborate on this numbers?