Tracking down the cause of locked servers

Yahoo Archive
1
Hi

Interesting.

We installed Process Explorer from sysinternals. It showed us that the more memory we put in the more memory that Epicor used. It always locked up until we went through the 32GB barrier. It slowly consumes memory until it has used it all. Killing a dead PID usually releases a mass of memory, sometime up to 10GB. 

It also allows to view the activity of an individual PID. If there is no I\O etc we usually feel safe to kill it. We are also using SSD drives and multi-core processors but on win 2003 64-bit, If you have a virtualised server it might be worth trying more memory.

I suspect that there are probably many factors involved. We have about 70 users logged in and the database is sitting around 15GB but we had this problem from the start, 5 years ago when the database was lot smaller. All I know is that it worked.

The only caveat is a process we know definitely killed the system was "labour entry". When a job had a "job to job" element it would cause us problems, so much so we wrote our own custom labour entry screen.

Erwin
2

We are on Vantage 8.03.408a Progress.


We are experiencing locked servers appearing throughout the day.  I have been trying to track down the cause, but so far I have not been able to pin it down to any customization or BPM process.  All I have been able to do is to monitor the servers using the Explorer Tool and kill the locked process using the PID when it occurs.  If I don't, eventually we run out of available servers.


Does anyone have any advise on how I can track down what is causing the locked servers?


Steven G.

3

The PID is listed in some tabs of system monitor…Before you kill the PID, check there to see if you can find the PID.

 

From: vantage@yahoogroups.com [mailto:vantage@yahoogroups.com]
Sent: Tuesday, July 22, 2014 7:54 AM
To: vantage@yahoogroups.com
Subject: [Vantage] Tracking down the cause of locked servers

 

 

We are on Vantage 8.03.408a Progress.

 

We are experiencing locked servers appearing throughout the day.  I have been trying to track down the cause, but so far I have not been able to pin it down to any customization or BPM process.  All I have been able to do is to monitor the servers using the Explorer Tool and kill the locked process using the PID when it occurs.  If I don't, eventually we run out of available servers.

 

Does anyone have any advise on how I can track down what is causing the locked servers?

 

Steven G.

4

It has been a while since I used it, but....


There is a command you can run in the PROENV window called 'ProGetStack ####'


Enter the PID after the ProGetStack, and a file called 'protrace.####' will show up in your OpenEdge working directory(oe102a_wrk).  This will tell you what was using that PID at the time.



--Matt

5

Do not kill PIDs.

 

Killing a PID is generally a bad idea because depending on what you kill, you can potentially cause data corruption.

 

If you are going to kill one, you should check to see what it is doing.

 

You can check what a PID is doing by looking in the System Monitor to see if anything there is running it, you can also check the log files by searching UP from the bottom of the logs for that PID number, assuming you have verbose logging turned on.


Also, when you are investigating this, start with whatever PID has the oldest “Last Change” date and time, except for any where the last change is about the same as the last time you restarted the appservers completely.

 

 

 

From: vantage@yahoogroups.com [mailto:vantage@yahoogroups.com]
Sent: Tuesday, July 22, 2014 8:54 AM
To: vantage@yahoogroups.com
Subject: [Vantage] Tracking down the cause of locked servers

 




We are on Vantage 8.03.408a Progress.

 

We are experiencing locked servers appearing throughout the day.  I have been trying to track down the cause, but so far I have not been able to pin it down to any customization or BPM process.  All I have been able to do is to monitor the servers using the Explorer Tool and kill the locked process using the PID when it occurs.  If I don't, eventually we run out of available servers.

 

Does anyone have any advise on how I can track down what is causing the locked servers?

 

Steven G.




6
Thanks for the advice everyone.  After more investigation, it appears that the locked servers are being caused by reports being printed.  But it is random, different users, different reports.

Is there something in general for report printing I could check?

Steven G.


On Tuesday, July 22, 2014 9:45 AM, "'Ned' TechnoBabbly@... [vantage]" <vantage@yahoogroups.com> wrote:


  
<div id="ygrps-yiv-744129495yiv9380380505ygrp-text">
  
  
  <div class="ygrps-yiv-744129495yiv9380380505WordSection1"><div class="ygrps-yiv-744129495yiv9380380505MsoNormal"><span style="font-size:11.0pt;color:#1F497D;">Do not kill PIDs.</span></div><div class="ygrps-yiv-744129495yiv9380380505MsoNormal"><span style="font-size:11.0pt;color:#1F497D;"> &nbsp;</span></div><div class="ygrps-yiv-744129495yiv9380380505MsoNormal"><span style="font-size:11.0pt;color:#1F497D;">Killing a PID is generally a bad idea because depending on what you kill, you can potentially cause data corruption.</span></div><div class="ygrps-yiv-744129495yiv9380380505MsoNormal"><span style="font-size:11.0pt;color:#1F497D;"> &nbsp;</span></div><div class="ygrps-yiv-744129495yiv9380380505MsoNormal"><span style="font-size:11.0pt;color:#1F497D;">If you are going to kill one, you should check to see what it is doing.</span></div><div class="ygrps-yiv-744129495yiv9380380505MsoNormal"><span style="font-size:11.0pt;color:#1F497D;"> &nbsp;</span></div><div class="ygrps-yiv-744129495yiv9380380505MsoNormal"><span style="font-size:11.0pt;color:#1F497D;">You can check what a PID is doing by looking

in the System Monitor to see if anything there is running it, you can also check the log files by searching UP from the bottom of the logs for that PID number, assuming you have verbose logging turned on.


Also, when you are investigating this, start with whatever PID has the oldest “Last Change” date and time, except for any where the last change is about the same as the last time you restarted the appservers completely.
 
 
 
From: vantage@yahoogroups.com [mailto:vantage@yahoogroups.com]
Sent: Tuesday, July 22, 2014 8:54 AM
To: vantage@yahoogroups.com
Subject: [Vantage] Tracking down the cause of locked servers
 



We are on Vantage 8.03.408a Progress.
 
We are experiencing locked servers appearing throughout the day.  I have been trying to track down the cause, but so far I have not been able to pin it down to any customization or BPM process.  All I have been able to do is to monitor the servers using the Explorer Tool and kill the locked process using the PID when it occurs.  If I don't,
eventually we run out of available servers.
 
Does anyone have any advise on how I can track down what is causing the locked servers?
 
Steven G.



</div>
 


<div style="color:#fff;height:0;"></div></div>
#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505 -- #ygrps-yiv-744129495yiv9380380505ygrp-mkp { border:1px solid #d8d8d8;font-family:Arial;margin:10px 0;padding:0 10px;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-mkp hr {
border:1px solid #d8d8d8;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-mkp #ygrps-yiv-744129495yiv9380380505hd {
color:#628c2a;font-size:85%;font-weight:700;line-height:122%;margin:10px 0;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-mkp #ygrps-yiv-744129495yiv9380380505ads {
margin-bottom:10px;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-mkp .ygrps-yiv-744129495yiv9380380505ad {
padding:0 0;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-mkp .ygrps-yiv-744129495yiv9380380505ad p {
margin:0;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-mkp .ygrps-yiv-744129495yiv9380380505ad a {
color:#0000ff;text-decoration:none;}
#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-sponsor #ygrps-yiv-744129495yiv9380380505ygrp-lc {
font-family:Arial;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-sponsor #ygrps-yiv-744129495yiv9380380505ygrp-lc #ygrps-yiv-744129495yiv9380380505hd {
margin:10px 0px;font-weight:700;font-size:78%;line-height:122%;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-sponsor #ygrps-yiv-744129495yiv9380380505ygrp-lc .ygrps-yiv-744129495yiv9380380505ad {
margin-bottom:10px;padding:0 0;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505actions {
font-family:Verdana;font-size:11px;padding:10px 0;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505activity {
background-color:#e0ecee;float:left;font-family:Verdana;font-size:10px;padding:10px;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505activity span {
font-weight:700;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505activity span:first-child {
text-transform:uppercase;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505activity span a {
color:#5085b6;text-decoration:none;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505activity span span {
color:#ff7900;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505activity span .ygrps-yiv-744129495yiv9380380505underline {
text-decoration:underline;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 .ygrps-yiv-744129495yiv9380380505attach {
clear:both;display:table;font-family:Arial;font-size:12px;padding:10px 0;width:400px;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 .ygrps-yiv-744129495yiv9380380505attach div a {
text-decoration:none;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 .ygrps-yiv-744129495yiv9380380505attach img {
border:none;padding-right:5px;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 .ygrps-yiv-744129495yiv9380380505attach label {
display:block;margin-bottom:5px;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 .ygrps-yiv-744129495yiv9380380505attach label a {
text-decoration:none;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 blockquote {
margin:0 0 0 4px;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 .ygrps-yiv-744129495yiv9380380505bold {
font-family:Arial;font-size:13px;font-weight:700;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 .ygrps-yiv-744129495yiv9380380505bold a {
text-decoration:none;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 dd.ygrps-yiv-744129495yiv9380380505last p a {
font-family:Verdana;font-weight:700;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 dd.ygrps-yiv-744129495yiv9380380505last p span {
margin-right:10px;font-family:Verdana;font-weight:700;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 dd.ygrps-yiv-744129495yiv9380380505last p span.ygrps-yiv-744129495yiv9380380505yshortcuts {
margin-right:0;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 div.ygrps-yiv-744129495yiv9380380505attach-table div div a {
text-decoration:none;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 div.ygrps-yiv-744129495yiv9380380505attach-table {
width:400px;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 div.ygrps-yiv-744129495yiv9380380505file-title a, #ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 div.ygrps-yiv-744129495yiv9380380505file-title a:active, #ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 div.ygrps-yiv-744129495yiv9380380505file-title a:hover, #ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 div.ygrps-yiv-744129495yiv9380380505file-title a:visited {
text-decoration:none;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 div.ygrps-yiv-744129495yiv9380380505photo-title a, #ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 div.ygrps-yiv-744129495yiv9380380505photo-title a:active, #ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 div.ygrps-yiv-744129495yiv9380380505photo-title a:hover, #ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 div.ygrps-yiv-744129495yiv9380380505photo-title a:visited {
text-decoration:none;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 div#ygrps-yiv-744129495yiv9380380505ygrp-mlmsg #ygrps-yiv-744129495yiv9380380505ygrp-msg p a span.ygrps-yiv-744129495yiv9380380505yshortcuts {
font-family:Verdana;font-size:10px;font-weight:normal;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 .ygrps-yiv-744129495yiv9380380505green {
color:#628c2a;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 .ygrps-yiv-744129495yiv9380380505MsoNormal {
margin:0 0 0 0;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 o {
font-size:0;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505photos div {
float:left;width:72px;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505photos div div {
border:1px solid #666666;height:62px;overflow:hidden;width:62px;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505photos div label {
color:#666666;font-size:10px;overflow:hidden;text-align:center;white-space:nowrap;width:64px;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505reco-category {
font-size:77%;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505reco-desc {
font-size:77%;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 .ygrps-yiv-744129495yiv9380380505replbq {
margin:4px;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-actbar div a:first-child {
margin-right:2px;padding-right:5px;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-mlmsg {
font-size:13px;font-family:Arial, helvetica, clean, sans-serif;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-mlmsg table {
font-size:inherit;font:100%;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-mlmsg select, #ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 input, #ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 textarea {
font:99% Arial, Helvetica, clean, sans-serif;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-mlmsg pre, #ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 code {
font:115% monospace;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-mlmsg * {
line-height:1.22em;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-mlmsg #ygrps-yiv-744129495yiv9380380505logo {
padding-bottom:10px;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-msg p a {
font-family:Verdana;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-msg p#ygrps-yiv-744129495yiv9380380505attach-count span {
color:#1E66AE;font-weight:700;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-reco #ygrps-yiv-744129495yiv9380380505reco-head {
color:#ff7900;font-weight:700;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-reco {
margin-bottom:20px;padding:0px;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-sponsor #ygrps-yiv-744129495yiv9380380505ov li a {
font-size:130%;text-decoration:none;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-sponsor #ygrps-yiv-744129495yiv9380380505ov li {
font-size:77%;list-style-type:square;padding:6px 0;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-sponsor #ygrps-yiv-744129495yiv9380380505ov ul {
margin:0;padding:0 0 0 8px;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-text {
font-family:Georgia;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-text p {
margin:0 0 1em 0;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-text tt {
font-size:120%;}

#ygrps-yiv-744129495 #ygrps-yiv-744129495yiv9380380505 #ygrps-yiv-744129495yiv9380380505ygrp-vital ul li:last-child {
border-right:none !important;}
#ygrps-yiv-744129495



7

Make sure you do not have any wide open reports (date range can go back forever for example)…I expect something could also happen if user puts in newest date before oldest date when report calls for the opposite too.  I know we have this happen now and then and I see one report more often than others, but I have checked the xml and the date range seems OK.  It just still happens at times…We have a separate report appserver and restart services if needed without restarting the main appserver.

 

From: vantage@yahoogroups.com [mailto:vantage@yahoogroups.com]
Sent: Tuesday, July 22, 2014 1:38 PM
To: vantage@yahoogroups.com
Subject: Re: [Vantage] Tracking down the cause of locked servers

 

 

Thanks for the advice everyone.  After more investigation, it appears that the locked servers are being caused by reports being printed.  But it is random, different users, different reports.

 

Is there something in general for report printing I could check?

 

Steven G.

 

On Tuesday, July 22, 2014 9:45 AM, "'Ned' TechnoBabbly@... [vantage]" <vantage@yahoogroups.com> wrote:

 

 

Do not kill PIDs.

 

Killing a PID is generally a bad idea because depending on what you kill, you can potentially cause data corruption.

 

If you are going to kill one, you should check to see what it is doing.

 

You can check what a PID is doing by looking in the System Monitor to see if anything there is running it, you can also check the log files by searching UP from the bottom of the logs for that PID number, assuming you have verbose logging turned on.


Also, when you are investigating this, start with whatever PID has the oldest “Last Change” date and time, except for any where the last change is about the same as the last time you restarted the appservers completely.

 

 

 

From: vantage@yahoogroups.com [mailto:vantage@yahoogroups.com]
Sent: Tuesday, July 22, 2014 8:54 AM
To: vantage@yahoogroups.com
Subject: [Vantage] Tracking down the cause of locked servers

 



We are on Vantage 8.03.408a Progress.

 

We are experiencing locked servers appearing throughout the day.  I have been trying to track down the cause, but so far I have not been able to pin it down to any customization or BPM process.  All I have been able to do is to monitor the servers using the Explorer Tool and kill the locked process using the PID when it occurs.  If I don't, eventually we run out of available servers.

 

Does anyone have any advise on how I can track down what is causing the locked servers?

 

Steven G.



 

8
Hi

We used to experience this all the time, we have upgraded the server and bumped the RAM up to 50GB with 8 cpu and have reduced the problem to minimal impact. We occasionally find a locked PID in the appserver, if it is over 3 hours old we just kill the PID and have had no problems so far. 

We tried to track the source but once the process is launched there appears to be no track back to the "thing" that launched it. We tried all sorts of audits and traces but found that the running "hung" process didn't seem to what had launched it.

Reports can cause a problem if you use open-ended date ranges e.g. from Today until forever, a end date makes it work better, also applies to scheduled tasks like MRP, it works better with an end date.

We suspect a lot are caused by end users launching a process or print then logging out or switching off.

Erwin
9
We have 32 GB Ram and two processors with a total of 24 cores, SSD drives on a raid 10 controller. Windows Server 2008 R2 Enterprise.  I don't think its the hardware and we didn't experience this problem before.  Some of the locked processes are from some things as simply printing a packing slip.

Steven G.


On Thursday, July 31, 2014 7:56 AM, "erwinjw@... [vantage]" <vantage@yahoogroups.com> wrote:


  
<div id="ygrps-yiv-143749373yiv2183326200ygrp-text">
  
  
  <div>Hi</div><div><br clear="none"></div><div>We used to experience this all the time, we have upgraded the server and bumped the RAM up to 50GB with 8 cpu and have reduced the problem to minimal impact. We occasionally find a locked PID in the appserver, if it is over 3 hours old we just kill the PID and have had no problems so far.&nbsp;</div><div><br clear="none"></div><div>We tried to track the source but once the process is launched there appears to be no track back to the &quot;thing&quot; that launched it. We tried all sorts of audits and traces but found that the running &quot;hung&quot; process didn&#39;t seem to what had launched it.</div><div><br clear="none"></div><div>Reports can cause a problem if you use open-ended date ranges e.g. from Today until forever, a end date makes it work better, also applies to scheduled tasks like MRP, it works better with an end date.</div><div><br clear="none"></div><div>We suspect a lot are caused by end users launching a

process or print then logging out or switching off.


Erwin

</div>
 


<div style="color:#fff;height:0;"></div></div>
#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 -- #ygrps-yiv-143749373yiv2183326200ygrp-mkp { border:1px solid #d8d8d8;font-family:Arial;margin:10px 0;padding:0 10px;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-mkp hr {
border:1px solid #d8d8d8;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-mkp #ygrps-yiv-143749373yiv2183326200hd {
color:#628c2a;font-size:85%;font-weight:700;line-height:122%;margin:10px 0;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-mkp #ygrps-yiv-143749373yiv2183326200ads {
margin-bottom:10px;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-mkp .ygrps-yiv-143749373yiv2183326200ad {
padding:0 0;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-mkp .ygrps-yiv-143749373yiv2183326200ad p {
margin:0;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-mkp .ygrps-yiv-143749373yiv2183326200ad a {
color:#0000ff;text-decoration:none;}
#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-sponsor #ygrps-yiv-143749373yiv2183326200ygrp-lc {
font-family:Arial;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-sponsor #ygrps-yiv-143749373yiv2183326200ygrp-lc #ygrps-yiv-143749373yiv2183326200hd {
margin:10px 0px;font-weight:700;font-size:78%;line-height:122%;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-sponsor #ygrps-yiv-143749373yiv2183326200ygrp-lc .ygrps-yiv-143749373yiv2183326200ad {
margin-bottom:10px;padding:0 0;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200actions {
font-family:Verdana;font-size:11px;padding:10px 0;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200activity {
background-color:#e0ecee;float:left;font-family:Verdana;font-size:10px;padding:10px;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200activity span {
font-weight:700;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200activity span:first-child {
text-transform:uppercase;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200activity span a {
color:#5085b6;text-decoration:none;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200activity span span {
color:#ff7900;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200activity span .ygrps-yiv-143749373yiv2183326200underline {
text-decoration:underline;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 .ygrps-yiv-143749373yiv2183326200attach {
clear:both;display:table;font-family:Arial;font-size:12px;padding:10px 0;width:400px;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 .ygrps-yiv-143749373yiv2183326200attach div a {
text-decoration:none;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 .ygrps-yiv-143749373yiv2183326200attach img {
border:none;padding-right:5px;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 .ygrps-yiv-143749373yiv2183326200attach label {
display:block;margin-bottom:5px;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 .ygrps-yiv-143749373yiv2183326200attach label a {
text-decoration:none;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 blockquote {
margin:0 0 0 4px;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 .ygrps-yiv-143749373yiv2183326200bold {
font-family:Arial;font-size:13px;font-weight:700;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 .ygrps-yiv-143749373yiv2183326200bold a {
text-decoration:none;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 dd.ygrps-yiv-143749373yiv2183326200last p a {
font-family:Verdana;font-weight:700;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 dd.ygrps-yiv-143749373yiv2183326200last p span {
margin-right:10px;font-family:Verdana;font-weight:700;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 dd.ygrps-yiv-143749373yiv2183326200last p span.ygrps-yiv-143749373yiv2183326200yshortcuts {
margin-right:0;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 div.ygrps-yiv-143749373yiv2183326200attach-table div div a {
text-decoration:none;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 div.ygrps-yiv-143749373yiv2183326200attach-table {
width:400px;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 div.ygrps-yiv-143749373yiv2183326200file-title a, #ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 div.ygrps-yiv-143749373yiv2183326200file-title a:active, #ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 div.ygrps-yiv-143749373yiv2183326200file-title a:hover, #ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 div.ygrps-yiv-143749373yiv2183326200file-title a:visited {
text-decoration:none;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 div.ygrps-yiv-143749373yiv2183326200photo-title a, #ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 div.ygrps-yiv-143749373yiv2183326200photo-title a:active, #ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 div.ygrps-yiv-143749373yiv2183326200photo-title a:hover, #ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 div.ygrps-yiv-143749373yiv2183326200photo-title a:visited {
text-decoration:none;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 div#ygrps-yiv-143749373yiv2183326200ygrp-mlmsg #ygrps-yiv-143749373yiv2183326200ygrp-msg p a span.ygrps-yiv-143749373yiv2183326200yshortcuts {
font-family:Verdana;font-size:10px;font-weight:normal;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 .ygrps-yiv-143749373yiv2183326200green {
color:#628c2a;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 .ygrps-yiv-143749373yiv2183326200MsoNormal {
margin:0 0 0 0;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 o {
font-size:0;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200photos div {
float:left;width:72px;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200photos div div {
border:1px solid #666666;height:62px;overflow:hidden;width:62px;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200photos div label {
color:#666666;font-size:10px;overflow:hidden;text-align:center;white-space:nowrap;width:64px;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200reco-category {
font-size:77%;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200reco-desc {
font-size:77%;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 .ygrps-yiv-143749373yiv2183326200replbq {
margin:4px;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-actbar div a:first-child {
margin-right:2px;padding-right:5px;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-mlmsg {
font-size:13px;font-family:Arial, helvetica, clean, sans-serif;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-mlmsg table {
font-size:inherit;font:100%;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-mlmsg select, #ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 input, #ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 textarea {
font:99% Arial, Helvetica, clean, sans-serif;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-mlmsg pre, #ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 code {
font:115% monospace;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-mlmsg * {
line-height:1.22em;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-mlmsg #ygrps-yiv-143749373yiv2183326200logo {
padding-bottom:10px;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-msg p a {
font-family:Verdana;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-msg p#ygrps-yiv-143749373yiv2183326200attach-count span {
color:#1E66AE;font-weight:700;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-reco #ygrps-yiv-143749373yiv2183326200reco-head {
color:#ff7900;font-weight:700;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-reco {
margin-bottom:20px;padding:0px;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-sponsor #ygrps-yiv-143749373yiv2183326200ov li a {
font-size:130%;text-decoration:none;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-sponsor #ygrps-yiv-143749373yiv2183326200ov li {
font-size:77%;list-style-type:square;padding:6px 0;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-sponsor #ygrps-yiv-143749373yiv2183326200ov ul {
margin:0;padding:0 0 0 8px;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-text {
font-family:Georgia;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-text p {
margin:0 0 1em 0;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-text tt {
font-size:120%;}

#ygrps-yiv-143749373 #ygrps-yiv-143749373yiv2183326200 #ygrps-yiv-143749373yiv2183326200ygrp-vital ul li:last-child {
border-right:none !important;}
#ygrps-yiv-143749373